madparts-rs 1.0 release

History

Back in Februari 2013 then coworker Romain S. showed me the new trend of programming editors that do continuous compilation while you type, showing you immediate feedback on your code. In parallel I also worked on 3D modeling for my 3D printer using the OpenSCAD program. OpenSCAD works by writing code in its custom language and then have it rendered.

An example:

openSCAD image

I had this idea of combining these two approaches to make an electronics footprint generator. And so the development of the original madparts program started.

To go quick I decided to write the program in python, but I wanted a compact language for the footprints so there I decided to use coffeescript, a functional javascript like language that compiles to javascript. Also I used openGL shading language for most of the drawing, because I found it interesting and hadn't touched it before.

Several changes were made among the way, and end of March 2013 a first release was made. It supported both Kicad and Eagle and linux, mac and windows.

After this a command line version of the program was added, and Debian packaging and 1.1 was released in May 2013.

In August 2013 1.2 was released which added support for the then brand new symbolic expressions based file format in Kicad.

In August 2015 the 2.0 release was done with mostly bugfixes and an update to the file format, but it also completely removed the whole file management system which existed, simplifying the program to just work on one file.

At this point the program pretty much did all I needed so further development stalled except for some minor bugfixes.

madparts image

Rust rewrite

In August 2016 I had been playing with the then pretty new rust programming language and decided a rewrite in it and simplifying the program even further would be fun to do.

The following were my initial goals:

Some brief searching showed that Qt (which I used in the python version) support was limited at best but GTK+ and cairo support seemed quite good with reasonable APIs (gtk-rs), and python support seemed to be ok with PyO3.

Practical

file monitoring

By monitoring the file being edited by an external editor via inotify the program can see when a change is saved and render the file. There is a practical caveat there though: most editors actually write the content to a temporary file and move that file over the old file on save (to avoid loss on case of system crash or power loss). This means in practice instead of monitoring the file, you have to monitor the containing directory.

    let _file_watch = ino.add_watch(
        &filedir,
        WatchMask::CREATE | WatchMask::MOVED_TO | WatchMask::CLOSE_WRITE,
    ).unwrap();

python interfacing

Using PyO3 running a python interpreter inside of rust is pretty straightforward. The biggest issue I ran into was dealing with the error situations:

While it is possible to get errors out of python into rust this is tedious and verbose. After some testing I came with a simple solution: do it all in python, providing python the filename to process, and when it fails, capture this in python as well and convert it in a simple Error object meaning from the rust perspective the python code always succeeds, it just has to check for this error object and display then contained message it when it is there instead of drawing the footprint.

def handle_load_python(filename):
    try:
        exec(open(filename).read(), globals(), globals())
        return flatten(footprint())
    except:
        import sys, traceback
        exc_type, exc_value, exc_traceback = sys.exc_info()
        message = "".join(traceback.format_exception(exc_type, exc_value, exc_traceback))
        e = PythonError(message)
        return [e]

rendering

Using Cairo rendering is pretty straightforward as well. A few traits on the Element type make it work:

trait BoundingBox {
    fn bounding_box(&self) -> Bound;
}

pub trait DrawElement {
    fn draw_element(&self, &cairo::Context, layer: Layer);
}

BoundingBox calculates the Bound of an element. By knowing all the bounds and combining them the program can automatically scale the drawing canvas correctly.

DrawElement allows an element to draws on a certain Layer. This is called Layer by layer for each element to have proper z-axis stacking of the drawings.

openSCAD image

KLC

KLC is supported by just executing the python KLC tool check_kicad_mod.py. The result is displayed in the KLC tab.

openSCAD image

exporting

Exporting just saves the footprint as a .kicad_mod file for usage in Kicad.

Release

This is the release 1.0 and the first public release. This means the program works, but is still far from feature complete. I'm adding more features as I need them for footprints. Documentation is not available yet, but I suggest looking at the examples in the footprint/ subdirectory or look in src/prelude.py directly to see what is supported by the python format.

For now the program is only tested in Linux but it should also run with perhaps minimal changes on OSX or Windows. I'm always happy to get a pull request for that on github.

Further plans

More features will be added as needed. Other things planned:

rust serde diesel table column limit

Problem

I have the following sql table:

CREATE TABLE system (
  id SERIAL PRIMARY KEY,
  name VARCHAR NOT NULL UNIQUE,
  allegiance_id SERIAL REFERENCES allegiance (id),
  state_id SERIAL REFERENCES state (id),
  government_id SERIAL REFERENCES government (id),
  security_id SERIAL REFERENCES security (id),
  needs_permit BOOLEAN DEFAULT FALSE,
  power_state_id SERIAL REFERENCES power_state (id),
  x DOUBLE PRECISION NOT NULL,
  y DOUBLE PRECISION NOT NULL,
  z DOUBLE PRECISION NOT NULL,
  simbad_ref VARCHAR DEFAULT '',
  controlling_minor_faction_id SERIAL, -- TODO REF
  reserve_type_id SERIAL REFERENCES reserve_type (id),
  is_populated BOOLEAN DEFAULT FALSE,
  edsm_id SERIAL,
  updated_at TIMESTAMPTZ
);

Running diesel print-schema > src/schema.rs generates the following schema code:

table! {
    system (id) {
        id -> Int4,
        name -> Varchar,
        allegiance_id -> Int4,
        state_id -> Int4,
        government_id -> Int4,
        security_id -> Int4,
        needs_permit -> Nullable<Bool>,
        power_state_id -> Int4,
        x -> Float8,
        y -> Float8,
        z -> Float8,
        simbad_ref -> Nullable<Varchar>,
        controlling_minor_faction_id -> Int4,
        reserve_type_id -> Int4,
        is_populated -> Nullable<Bool>,
        edsm_id -> Int4,
        updated_at -> Nullable<Timestamptz>,
    }
}

Compiling this gives the following error among things:

error[E0277]: the trait bound `(schema::system::columns::id, schema::system::columns::name, schema::system::columns::allegiance_id, schema::system::columns::state_id, schema::system::columns::government_id, schema::system::columns::security_id, schema::system::columns::needs_permit, schema::system::columns::power_state_id, schema::system::columns::x, schema::system::columns::y, schema::system::columns::z, schema::system::columns::simbad_ref, schema::system::columns::controlling_minor_faction_id, schema::system::columns::reserve_type_id, schema::system::columns::is_populated, schema::system::columns::edsm_id, schema::system::columns::updated_at): diesel::Expression` is not satisfied
  --> src/schema.rs:58:1
   |
58 | / table! {
59 | |     system (id) {
60 | |         id -> Int4,
61 | |         name -> Varchar,
...  |
77 | |     }
78 | | }

This is a bit unclear and I first thought that perhaps there is an issue with the Timestamptz type as commenting out that field makes it compiles fine. However further testing showed that commenting out any field solves it.

Solution

diesel has a default table column limit of 16. Our table has 17 columns. Adding large-tables to Cargo.toml solves it.

[dependencies]
diesel = { version = "1.0", features = ["postgres", "chrono", "large-tables"] }

Thinking about it this way the error actually makes sense. Diesel has no implementation for diesel::Expression for a 17 column table (which is implemented as a 17 size tuple).

rust serde deserialization of an enum variant

Intro

For a program I'm working on I have this datastructure:

pub enum State {
    None,
    Expansion,
    War,
    CivilWar,
    ...
}

This same datastructure is returned from different external JSON API's where the formatting is slightly different. I'm using serde and serde_json for deserialization. Without any special processing the following program will deserialize "CivilWar" to State::CivilWar:

#[macro_use]
extern crate serde_derive;
extern crate serde_json;

#[derive(Debug, Deserialize)]
pub enum State {
    None,
    Expansion,
    War,
    CivilWar,
    ...
}

fn main() {
    let s = r#" "CivilWar" "#;
    let c:State = serde_json::from_str(s).unwrap();
    println!("input: {} output: State::{:?}", s, c);
}

This will output: input: "CivilWar" output: State::CivilWar.

Lowercase

The JSON format I'm deserialiazing from actually specifies the state as lowercase. This is easily accomodated by adding an annotation #[serde(rename_all = "lowercase")] to the enum:

#[derive(Debug, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum State {
    None,
    Expansion,
    War,
    CivilWar,
    ...
}

Now "civilwar" will be deserialized as State::CivilWar. Of course "CivilWar" won't deserialize anymore.

Space

However some files contain "civil war" with a space in between. This will still not be mapped correctly. As we have multiple possible inputs, a simple rename will no longer suffice.

A custom implementation of Deserialize works, but is a lot of boilerplate code:

#[derive(Debug)]
pub enum State {
    None,
    Expansion,
    War,
    CivilWar,
    ...
}

impl<'de> Deserialize<'de> for State {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: Deserializer<'de>,
    {
        let s = String::deserialize(deserializer)?.to_lowercase();
        let state = match s.as_str() {
            "none" => State::None,
            "expansion" => State::Expansion,
            "war" => State::War,
            "civil war" | "civilwar" => State::CivilWar,
            ...
            other => { return Err(de::Error::custom(format!("Invalid state '{}'", other))); },
        };
        Ok(state)
    }
}

Variant deserialize_with

In principle it should be possible to make a custom deserialization function only for the offending variants (State::CivilWar and State::CivilUnrest) by introducing a variant annotation like this:

#[derive(Debug, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum State {
    None,
    Expansion,
    War,
    #[serde(deserialize_with = "de_civilwar")]
    CivilWar,
    Election,
    Boom,
    Bust,
    CivilUnrest,
    Famine,
    Outbreak,
    Lockdown,
    Investment,
    Retreat,
}

fn de_civilwar<'de, D>(deserializer:D)-> Result<(), D::Error>
    where D: Deserializer<'de> {
    let s = String::deserialize(deserializer)?.to_lowercase();
    println!("found: {}", s);
    if s.as_str() == "civilwar" || s.as_str() == "civil war" {
        Ok(())
    } else {
        Err(
            de::Error::invalid_value(
                Unexpected::Str(&s),
                &r#""civil war" or "civilwar""#
            )
        )
    }
}

However using this fails with an error: invalid type: unit variant, expected newtype variant. At this point it is unclear to my why this doesn't work as it matches the documentation. To narrow it down I implemented a variant of the problem based on the test contained in serde:

#[macro_use]
extern crate serde_derive;
extern crate serde_json;
extern crate serde;

use serde::de::{self, Deserialize, Deserializer, Unexpected};

#[derive(Debug, PartialEq, Serialize, Deserialize)]
enum WithVariant {
    #[serde(deserialize_with = "deserialize_u8_as_unit_variant")]
    Unit,
}

fn deserialize_u8_as_unit_variant<'de, D>(deserializer: D) -> Result<(), D::Error>
where
    D: Deserializer<'de>,
{
    let n = u8::deserialize(deserializer)?;
    if n == 0 {
        Ok(())
    } else {
       Err(de::Error::invalid_value(Unexpected::Unsigned(n as u64), &"0"))
    }
}

fn main() {
    let s1 = "0";
    let i:u8 = serde_json::from_str(s1).unwrap();
    println!("i: {}", i);

    
    let s = "0";
    let c:WithVariant = serde_json::from_str(s).unwrap();
    println!("input: {} output: {:?}", s, c);
}

This fails in a different way, with the error: ExpectedSomeValue, line: 1, column: 1.

Either I'm overlooking something or there is a bug in the libraries.

Update

After some help from David Tolnay one of authors of serde, it turns out that the enum variant deserialize_with feature is meant to be used in a different way.

For the example above from the testcase this works:

    let s = r#"{ "Unit": 0 }"#;
    let c:WithVariant = serde_json::from_str(s).unwrap();
    println!("input: {} output: {:?}", s, c);

meaning the variant needs to be contained in another structure.

Finally David offered the following elegant alternative:

use serde::de::{Deserialize, Deserializer, IntoDeserializer};

#[derive(Debug, Deserialize)]
#[serde(rename_all = "lowercase")]
#[serde(remote = "State")]
pub enum State {
    Expansion,
    CivilWar,
    /* ... */
}

impl<'de> Deserialize<'de> for State {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
        where D: Deserializer<'de>
    {
        let s = String::deserialize(deserializer)?;
        if s == "civil war" {
            Ok(State::CivilWar)
        } else {
            State::deserialize(s.into_deserializer())
        }
    }
}

Which provides the special handling but avoids the boilerplate for the common cases.

All the example code used in this blog past can be found here.

fixing Hue persistence using async IO in Rust - Futures

Intro

I have two Philips Hue lamps and mostly like them when they are in the very warm white (2000k) color temperature setting.

The problem is that the setting does not persist when the lamps are switched using a physical power switch. To solve this I wrote the Hue Persistence program last year.

This program runs on the LAN and basically polls and stores the lamp settings and when it it detects a lamp going from off to on it restores the last setting.

After using it for a few weeks it became clear that there is one big problem with it: the Philips Hue hub sometimes accepts a connection from my program but then does not respond, yet keeps the connection open. As the program is written using synchronous IO, this causes the program to hang forever, and it doesn't work anymore.

Solution

The Hue Persistence program uses the Rust Philips Hue library which using Hyper as a HTTP client to talk to the Philips Hue hub via it's restful API. This library still uses Hyper 0.10 which does not provide async IO yet.

Recently there has been a lot of work for asynchronous IO in rust using Tokio and Futures. Version 0.11 of Hyper uses this.

First thing I had to do is port the Rust Philips Hue library to use Hyper 0.11 and thus async IO. This turned out to be quite an undertaking. The result of this can be found on the hyper_0.11 branch of my fork of the library. It is still a bit ugly, but works.

So far I've found using Futures for async IO to have the following issues:

error handling can be very confusing; error types need to be the same in future composition and you have to explicitly do error type conversions to make things work future composition results in complex types, returning to Box<Future> helps a bit, but it still can be rather confusing (I didn't try the not yet released impl Trait solution) it's not so obvious which future composition methods to use when, docs are sparse branching (if, ...) inside futures is tedious again because of types having to match perfectly In the end though I got the satisfaction of a working async library, after which I updated the Hue Persistence app to have a timeout on it's requests. This again turned out to be really tedious. I had to use select2 with a sleep instead of tokio_timer::Timer::timeout() because it was just impossible to get the types to work for the error types when using tokio_timer::Timer, due to the TimeoutError containing Future type as a containing type in the signature of timeout().

Conclusion

My lamps are behaving fine now!

Async IO in Rust using Futures/tokio is interesting but at this point it still feels like "getting the types right" is somewhat in the way of normal coding productivity. Some ergonomic improvements/language support could surely be used. To be revisited in the Future (pun intended ;) ).

implementing a simple language switch using rocket in rust

Let's look how easy it is to implement a simple cookie based language switch in the rocket web framework for the rust programming language. Defining the language type:

#[derive(PartialEq)]
enum Lang {
    Nl,
    En,
}

In this case there will be support for Dutch and English. PartialEq is derived to be able to compare Lang items with ==.

The Default trait is implemented to define the default language:

impl Default for Lang {
    fn default() -> Lang {
        Lang::Nl
    }
}

The FromStr trait is implemented to allow creating a Lang item from a string.

impl FromStr for Lang {
    type Err = Error;
    fn from_str(s:&str) -> Result<Self> {
        match s {
            "nl" => Ok(Lang::Nl),
            "en" => Ok(Lang::En),
            o => Err(format!("Unsupported language: {}", o).into()),
        }
    }
}

The Into<&'static str> trait is added to allow the conversion in the other direction.


impl Into<&'static str> for Lang {
    fn into(self) -> &'static str {
        match self {
            Lang::Nl => "nl",
            Lang::En => "en",
        }
    }
}

Finally the FromRequest trait is implemented to allow extracting the "lang" cookie from the request.

impl<'a, 'r> FromRequest<'a, 'r> for Lang {
    type Error = ();

    fn from_request(request: &'a Request<'r>) -> Outcome<Lang, ()> {
        match request.cookies().get_private("lang") {
            Some(r) => {
                match Lang::from_str(r.value()) {
                    Ok(l) => Success(l),
                    Err(_) => Success(Lang::default()),
                }
            }
            None => Success(Lang::default()),
        }
    }
}

It always succeeds and falls back to the default when no cookie or an unknown language is is found. How to use the Lang constraint on a request:

#[get("/page")]
fn page(lang: Lang) -> content::HTML<String> {
    let hello = if lang == Lang::Nl {
       "Hallo daar!"
    } else {
       "Hello there!"
    };
    content::HTML(format!(
"<html>
  <body>
   <h1>{}</h1>
   <a href='/lang/en'>English</a>
   <a href='/lang/nl'>Nederlands</a>
  </body>
</html>", hello))
}

And the language switch page:

#[get("/lang/<lang>")]
fn lang(mut cookies: Cookies, lang: String) -> Result<Redirect> {
    let lang:&'static str = Lang::from_str(&lang)?.into();
    info!("Setting language to: {}", lang);
    cookies.add_private(Cookie::new("lang", lang));
    Ok(Redirect::to("/page"))
}

And as a cherry on the pie, let's have the language switch page automatically redirect to the referrer. First let's implement a FromRequest trait for our own Referer type:

struct Referer {
    url:String
}

impl<'a, 'r> FromRequest<'a, 'r> for Referer {
    type Error = ();

    fn from_request(request: &'a Request<'r>) -> Outcome<Referer, ()> {
        match request.headers().get_one("Referer") {
            Some(r) => Success(Referer { url:r.into() }),
            None => Forward(()),
        }
    }
}

When it finds a Referer header it uses the content, else the request is forwarded to the next handler. This means that if the request has no Referer header it is not handled, and a 404 will be returned. Finally let's update the language switch request handler:


#[get("/lang/<lang>")]
fn lang(mut cookies: Cookies, referer: Referer, lang: String) -> Result<Redirect> {
    let lang:&'static str = Lang::from_str(&lang)?.into();
    info!("Setting language to: {}", lang);
    cookies.add_private(Cookie::new("lang", lang));
    Ok(Redirect::to(&referer.url))
}

Pretty elegant. A recap with all code combined and adding the missing glue:

#![feature(plugin)]
#![plugin(rocket_codegen)]

extern crate rocket;
#[macro_use]
extern crate log;

use rocket::request::{Outcome, Request, FromRequest};
use rocket::outcome::Outcome::*;
use rocket::response::{NamedFile, Redirect, content};
use rocket::http::{Cookie, Cookies};

use std::str::FromStr;

#[derive(PartialEq)]
enum Lang {
    Nl,
    En,
}

impl Default for Lang {
    fn default() -> Lang {
        Lang::Nl
    }
}

impl FromStr for Lang {
    type Err = Error;
    fn from_str(s:&str) -> Result<Self> {
        match s {
            "nl" => Ok(Lang::Nl),
            "en" => Ok(Lang::En),
            o => Err(format!("Unsupported language: {}", o).into()),
        }
    }
}

impl Into<&'static str> for Lang {
    fn into(self) -> &'static str {
        match self {
            Lang::Nl => "nl",
            Lang::En => "en",
        }
    }
}

impl<'a, 'r> FromRequest<'a, 'r> for Lang {
    type Error = ();

    fn from_request(request: &'a Request<'r>) -> Outcome<Lang, ()> {
        match request.cookies().get_private("lang") {
            Some(r) => {
                match Lang::from_str(r.value()) {
                    Ok(l) => Success(l),
                    Err(_) => Success(Lang::default()),
                }
            }
            None => Success(Lang::default()),
        }
    }
}

struct Referer {
    url:String
}

impl<'a, 'r> FromRequest<'a, 'r> for Referer {
    type Error = ();

    fn from_request(request: &'a Request<'r>) -> Outcome<Referer, ()> {
        match request.headers().get_one("Referer") {
            Some(r) => Success(Referer { url:r.into() }),
            None => Forward(()),
        }
    }
}

#[get("/page")]
fn page(lang: Lang) -> content::HTML<String> {
    let hello = if lang == Lang::Nl {
       "Hallo daar!"
    } else {
       "Hello there!"
    };
    content::HTML(format!(
"<html>
  <body>
   <h1>{}</h1>
   <a href='/lang/en'>English</a>
   <a href='/lang/nl'>Nederlands</a>
  </body>
</html>", hello))
}

#[get("/lang/<lang>")]
fn lang(mut cookies: Cookies, referer: Referer, lang: String) -> Result<Redirect> {
    let lang:&'static str = Lang::from_str(&lang)?.into();
    info!("Setting language to: {}", lang);
    cookies.add_private(Cookie::new("lang", lang));
    Ok(Redirect::to(&referer.url))
}

fn main() {
    rocket::ignite()
        .attach(Template::fairing())
        .mount("/", routes![lang , page])
        .launch();
}