Archive

EasyThreeD X1 Heated Bed Mod

(If you don't want to read this whole thing, skip to the end of the post for a tl;dr version)

I was lucky enough to get a Labists X1 3D printer for Christmas a few weeks ago, and it's the first 3D printer I've had or really even interacted with.

It's been a fascinating journey so far, learning about how to calibrate a printer, how to use slicers, and how to start making my own models.

Something that became obvious fairly quickly though, was that the printer would be more reliable with a heated bed. I've been able to get reliable prints via the use of rafts, but that adds time to prints and wastes filament, so I decided to see if I could mod the printer to have a heated bed.

I started Googling and quickly discovered that my printer is actually a rebadged EasyThreed X1 and that EasyThreed sell a hotbed accessory for the X1, but it's an externally powered/controlled device. That's fine in theory, but I have quickly gotten very attached to being able to completely remotely control the printer via Octoprint. So, the obvious next step was to try and mod the printer to be able to drive the heater directly.

Looking inside the controller box showed a pretty capable circuit board:

EasyThreed X1 controller board

but it was instantly obvious that next to the power terminal for the extruder heater, was a terminal labelled HOT-BED:

Hot bed power terminal

Next on my journey of discovery was the communication info that Octoprint was sending/receiving, among which I saw:

Recv: echo:Marlin 1.1.0-RC3

which quickly led me to the Marlin open source project which, crucially, is licensed as GPL. For those who don't know, GPL means that since Labists have given me a binary of Marlin in the printer, they have to give me the source code if I ask for it.

I reached out to Labists and they were happy to supply the source, then I also emailed EasyThreed to ask if I could have their source for the X1 as well (and while I was at it, their X3 printer, which looks a lot like the X1, but ships with a heated bed already as part of the product). They sent me the source with no real issues, so I grabbed the main Marlin repo, checked out the tag for 1.1.0-RC3 and started making branches for the various Labists/EasyThreed source trees I'd acquired. Since their changes were a bit gratuitous in places (random whitespace changes, DOS line endings, tabs, etc) I cleaned them up quite a bit to try and isolate the diffs to code/comment changes.

Since it's all GPL, I've republished their code with my cleanups:

The specific diffs aren't particularly important (although the Labists firmware does have some curious changes, like disabling thermal runaway protection), but by reading up a bit on configuring Marlin, and comparing the differences between the X3 and the X1, it seemed like very little would need to change to enable the bed heater and its temperature sensor (a header for which is also conveniently present on the controller board).

At this point in the investigation I had:

  • A controller board with:
    • A power terminal for a bed heater
    • A header for a bed temperature sensor
  • Source for the controller firmware
  • Source for an extremely similar printer that has a bed heater
  • An external bed heater with power and sensor cables

Not a bad situation to be in!

Diving into the firmware, I found that Marlin keeps most board-specific settings in Configuration.h and specifically, it contains #define TEMP_SENSOR_BED 0. The number that TEMP_SENSOR_BED is defined as, indicates to Marlin what type of temp sensor is attached (with 0 obviously meaning nothing is attached). The X3 has a value of 1 (a 100k thermistor), but I found that I could only get reliable readings with it set to 4 (a 10k thermistor).

Believe it or not, that's actually the only thing that has to change, but I did also change #define BED_MAXTEMP 150 because 150C seems kind of high. This define sets a temperature at which Marlin will shut itself down as a safety measure. As far as I can tell, 50C-70C is a more realistic range for PLA, and even with ABS it seems as though 110C is recommended. I haven't printed ABS yet and don't have any real plans to, so I reduced the safety limit to 100C. I also modified the build version strings in Default_Version.h so I'd be able to quickly tell in Octoprint if I had successfully uploaded a new firmware.

Next came the challenge of building the firmware. I grabbed the latest Arduino IDE, but it failed to compile Marlin correctly (perhaps because I was using the macOS version of Ardino IDE). Labists helpfully included a Windows build of Arduino IDE 1.0.5 with their firmware source, which was able to build it. Arduino IDE is also GPL, but I haven't republished that yet because I haven't audited the archive for other things that I don't have rights to distribute.

To get the firmware to upload correctly to the X1, I had to set the board type in Arduino IDE to Melzi and select the COM port for its USB interface, except its USB interface wasn't showing up and Windows' Device Manager couldn't find a driver for it. Some Googling for the USB VID/PID of that device led me to the manufacturer of the CH340 chipset and their drivers.

Finally the moment of truth - was I about to destroy a controller board with a bad firmware/driver? I clicked the Upload button, waited for it to complete, attached the controller to my Octoprint machine again and.......

Recv: echo:Marlin 1.1.0-RC3-cmsj

Success! I then waited for Octoprint to start communicating with the printer and monitoring temperatures...

Recv: ok T:24.2 /0.0 B:23.6 /0.0 T0:24.2 /0.0 @:0 B@:0

For those of you who aren't familiar with Octoprint/GCode, the T:24.2 is the temp sensor in the extruder and the B23.6 is the reading from the bed sensor! Another success!

After replacing the X1's 30W power supply with a 60W variant so it could power the motors and the heater, I asked it to heat up to 50C, and after a little while....

Recv: ok T:28.1 /0.0 B:49.9 /50.0 T0:28.1 /0.0 @:0 B@:127

Perfect!

And here is the first test print I did, to make sure everything else was still working:

Calibration cubes

The cubes on the left are from before the heated bed, where I was having to level the bed closer to the nozzle to get enough adhesion and the cube on the right is the first print with the heated bed. I think the results speak for themselves - much better detail retention. It's not visible, but the "elephant's foot" is gone too!

This has been a super rewarding journey, and I'm incredibly grateful to all the people in the 3D printing community upon whose shoulders I am standing. It's a rare and beautiful thing to find a varied community of products, projects and people, all working on the same goals and producing such high quality hardware and software along the way.

And now the tl;dr version

If you want to do this mod to your X1, here are some things you should know, and some things you will need:

  • I am not responsible for your printer. This is a physical and firmware mod, please be careful and think about what you're doing.
  • Buy the official hotbed accessory, open its control box and unplug the temperature sensor cable. If for some reason you use a different hotbed, it needs to be 12V, draw no more than 30W, and your temp sensor will need to be something that Marlin can understand via the TEMP_SENSOR_BED define.
  • Buy a 12V 5A barrel plug power supply (I used this one but there are a million options). Use this from now on to power your X1.
  • Grab the modified Marlin source from my GitHub repo:
    • Either EasyThreed X1 - see the precise changes from EasyThreed's firmware here
    • Or Labists X1 - this has more changes than the EasyThreed version, since I pulled back in some of Labists changes, but left thermal runaway protection enabled.
  • Install the CH340 USB Serial drivers. There seem to be lots of places to get these from, I used these
  • Install Arduino IDE 1.0.5 - still available from the bottom of this page
  • In Arduino IDE, open the Marlin.ino file from the Marlin directory and click the ✔ button on the toolbar, this will compile the source so you can check everything is installed correctly.
  • If you plan to print PLA, you might want to increase the BED_MAXTEMP define to something higher than 100.
  • Remove the bed-levelling screws from your X1, swap the original bed for the heated one.
  • Open the controller box of your X1, plug the bed's thermal sensor into the controller board in the TB1 header.
  • Wire the bed's power into the green HOT-BED terminal. For the best results you probably want to unsolder the original power cable from the bed and use something thinner and more flexible (but at the very least you need something longer).
  • Reassemble the controller box and run all the wires neatly. I recommend you manually move the bed around to make sure neither the power nor temp sensor wires snag on anything.
  • Connect the controller box's USB port to your PC, and in Arduino IDE click the ➡ button to compile and upload the firmware. Wait until it says Upload complete.
  • In theory, you're done! Check the temperature readings in some software that can talk to the printer (Octoprint, Pronterface, etc.), tell it to turn the bed heater on and make sure the temps rise to the level you asked for. I would definitely encourage you to do this while next to the printer, in case something goes dangerously wrong!

Update: Failing to create an app

Previously I wrote about how I'd tried to create an app, but ultimately failed because I wasn't getting the results I wanted out of the macOS CoreAudio APIs.

Thanks to some excellent input from David Lublin I refactored the code to be able to switch easily between different backend audio APIs, and implemented a replacement for CoreAudio using AVFoundation's AVCaptureSession and it seems to work!

I'd still like to settle back on CoreAudio at some point, but for now I can rest assured that whenever the older versions of SoundSource stop working, I still have a working option.


Overengineered email migration

I recently had the need to migrate someone in my family off an old ISP email account, onto a more modern email account, without simply shutting down the old account. The old address has been given out to people/companies for at least a decade, so it's simply not practical to stop receiving its email.

Initially, I used the ISP's own server-side filtering to forward emails on to the new account and then delete them, however, all of the fantabulous anti-spam technologies that are used these days, conspired to make it unreliable.

So instead, I decided that since I can access IMAP on both accounts, and I have a server at home running all the time, I'd just use some kind of local tool to fetch any emails that show up on the old account and move them to the new one.

After some investigation, I settled on imapsync as the most capable tool for the job. It's ultimately "just" a Perl script, but it's fantastically well maintained by Gilles Lamiral. It's Open Source, but I'm a big fan of supporting FOSS development, so I happily paid the 60€ Gilles asks for.

My strong preference these days is always to run my local services in Docker, and fortunately Gilles publishes an official imapsync Dockule so I set to work in Ansible to orchestrate all of the pieces I needed to get this running.

The first piece was a simple bash script that calls imapsync with all of the necessary command line options:

#!/bin/bash
# This is /usr/local/bin/imapsync-user-isp-fancyplace.sh
/usr/bin/docker run -u root --rm -v/root/.imap-pass-isp.txt:/isp-pass.txt -v/root/.imap-pass-fancyplace.txt:/fancyplace-pass.txt gilleslamiral/imapsync \
    imapsync \
        --host1 imap.isp.net --port1 993 --user1 olduser@isp.net --passfile1 /isp-pass.txt --ssl1 --sslargs1 SSL_verify_mode=1 \
        --host2 imap.fancyplace.com --port2 993 --user2 newuser@fancyplace.com --passfile2 /fancyplace-pass.txt --ssl2 --sslargs2 SSL_verify_mode=1 \
        --automap \
        --nofoldersizes --nofoldersizesatend \
        --delete1 --noexpungeaftereach \
        --expunge1

Please test this with the --dry option if you ever want to do this - the --automap option worked incredibly well for me (even translating between languages for folders like "Sent Messages"), but check that for yourself.

What this script will do is start a Docker container and run imapsync within it, which will then check all folders on the old IMAP server and sync any found emails over to the new IMAP server and then delete them from the old server. This is unfortunately necessary because the old ISP in question has a pretty low storage limit and I don't want future email flow to stop because we forgot to go and delete old emails. imapsync appears to be pretty careful about making sure an email has synced correctly before it deletes it from the old server, so I'm not super worried about data loss.

The IMAP passwords are read from files that live in /root/ on my server (with 0400 permissions) and they are mounted through into the container. For the new IMAP account, this is a "per-device" password rather than the main account password, so it won't change, and is easy to revoke.

This isn't a complete setup yet though, because after doing one sync, imapsync will exit and Docker will obey its --rm option and delete the container. What we now need is a regular trigger to run this script and while this used to mean cron, nowadays it could also mean a systemd timer. So, I created a simple systemd service file which gets written to /etc/systemd/system/imapsync-user-isp-fancyplace.service and enabled in systemd:

[Unit]
Description=User IMAP Sync
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/imapsync-user-isp-fancyplace.sh
Restart=no
TimeoutSec=120

and a systemd timer file which gets written to /etc/systemd/system/imapsync-user-isp-fancyplace.timer, and then both enabled and started in systemd:

[Unit]
Description=Trigger User IMAP Sync

[Timer]
Unit=imapsync-user-isp-fancyplace.service
OnUnitActiveSec=10min
OnUnitInactiveSec=10min
Persistent=true

[Install]
WantedBy=timers.target

This will trigger every 10 minutes and start the specified service, which executes the script that starts the Dockule to sync the email. Simple!

And just to show a useful command, you can check when the timer last triggered, and when it will trigger next, like this:

# systemctl list-timers
NEXT                         LEFT          LAST                         PASSED             UNIT                               ACTIVATES
Mon 2019-05-20 17:38:13 BST  27s left      Mon 2019-05-20 17:28:13 BST  9min ago           imapsync-user-isp-fancyplace.timer imapsync-user-isp-fancyplace.service
[snip unrelated timers]

9 timers listed.
Pass --all to see loaded but inactive timers, too.

Failing to create an app

I've just published https://github.com/cmsj/HotMic/ which contains a very good amount of a macOS app I had hoped to complete and sell for a couple of bucks on the Mac App Store.

However, I failed to get it working, primarily because I don't know enough of CoreAudio to get it working, and because I burned almost all of the time I had to write the app, fighting with things that, it turns out, were never going to work.

So, chalk that one up to experience I guess. Maybe the next person who has this idea will find my repo and spend their allotted time getting it to work :)

For the curious, the app's purpose was to be a Play Through mechanism for OS X. What is a Play Through app? It means the app reads audio from one device (e.g. a microphone or a Line In port) and writes it out to a different device (e.g. your normal speakers). This lets you use your Mac as a very limited audio mixer. I want it so the Line Out from my PC can be connected to my iMac - then all of my computer audio comes out of one set of speakers with one keyboard volume control setup.

For the super curious, I'd be happy to get back to working on the app if someone who knows more about Core Audio than I do, wants to get involved!


Abusing Gmail as a ghetto dashboard

I'm sure many of us receive regular emails from the same source - by which I mean things like a daily status email from a backup system, or a weekly newsletter from a blogger/journalist we like, etc.

These are a great way of getting notified or kept up to date, but every one of these you receive is also a piece of work you need to do, to keep your Inbox under control. Gmail has a lot of powerful filtering primitives, but as far as I am able to tell, none of them let you manage this kind of email without compromise.

My ideal scenario would be that, for example, my daily backup status email would keep the most recent copy in my Inbox, and automatically archive older ones. Same for newsletters - if I didn't read last week's one, I'm realistically never going to, so once it's more than a couple of weeks stale, just get it out of my Inbox.

Thankfully, Google has an indirect way of making this sort of thing work - Google Apps Script. You can trigger small JavaScript scripts to run every so often, and operate on your data in various Google apps, including Gmail.

So, I quickly wrote this script and it runs every few hours now:

// Configuration data
// Each config should have the following keys:
//  * age_min: maps to 'older_than:' in gmail query terms
//  * age_max: maps to 'newer_than:' in gmail query terms
//  * query: freeform gmail query terms to match against
//
// The age_min/age_max values don't need to exist, given the freeform query value,
// but age_min forces you to think about how frequent the emails are, and age_max
// forces you to not search for every single email tha matches the query
//
// TODO:
//  * Add a per-config flag that skips the archiving if there's only one matching thread (so the most recent matching email always stays in Inbox)
var configs = [
  { age_min:"14d", age_max:"90d", query:"subject:(Benedict's Newsletter)" },
  { age_min:"7d",  age_max:"30d", query:"from:hello@visualping.io subject:gnubert" },
  { age_min:"1d",  age_max:"7d",  query:"subject:(Nightly clone to Thunderbay4 Successfully)" },
  { age_min:"1d",  age_max:"7d",  query:"from:Amazon subject:(Arriving today)" },
  ];

function processInbox() {
  for (var config_key in configs) {
    var config = configs[config_key];
    Logger.log("Processing query: " + config["query"]);

    var threads = GmailApp.search("in:inbox " + config["query"] + " newer_than:" + config["age_max"] + " older_than:" + config["age_min"]);
    for (var thread_key in threads) {
      var thread = threads[thread_key];
      Logger.log("  Archiving: " + thread.getFirstMessageSubject());

      thread.markRead();
      thread.moveToArchive();
    }
  }
}

(apologies for the very basic JavaScript - it's not a language I have any real desire to be good at. Don't @ me).


Fixing an error in Xcode Instruments's Leaks profile

As part of our general effort to try and raise the quality of Hammerspoon, I've been working with @latenitefilms to track down some memory leaks, which can be very easy if you use the Leaks profile in Xcode's "Instruments" tool. I tried this various ways, but I kept running into this error:

Screenshot

After asking on the Apple Developer Forums we got an interesting response from an Apple employee that code signing might be involved. One change later to not do codesigning on Profile builds and Leaks is working again!

So there we go, if you see "An error occurred trying to capture Leaks data" and "Unable to acquire required task port", one thing to check is your code signing setup. I don't know what specifically was wrong, but it's easy enough to just not sign local debug/profile builds most of the time anyway.


AmigaOS 4.1 Final Edition in Qemu

So this is a fun one, some marvellous hackers, including Zoltan Balaton and Sebastien Mauer have been working on Qemu to add support for the Sam460ex motherboard, a PowerPC system from 2010. Of particular interest to me is that this was a board which received an official port of Amiga OS 4, the spiritual successor to AmigaOS, one of my very favourite operating systems.

I'll probably write more about this later, but for now, here is a simple screenshot of the install CD having just booted.

Update: Zoltan has published a page with information about how to get it working, see here

Screenshot


Home networking like a pro - Part 1.1 - Network Storage Redux

Back in this post I described having switched from a Mac Mini + DAS setup, to a Synology and an Intel NUC setup, for my file storage and server needs.

For a time it was good, but I found myself wanting to run more server daemons, and the NUC wasn't really able to keep up. The Synology was plodding along fine, but I made the decision to unify them all into a more beefy Linux machine.

So, I bought an AMD Ryzen 5 1600 CPU and an A320M motherboard, 16GB of RAM and a micro ATX case with 8 drive bays, and set to work. That quickly proved to be a disaster because Linux wasn't stable on the AMD CPU - I hadn't even thought to check, because why wouldn't Linux be stable on an x86_64 CPU in 2018?! With that lesson learned, I swapped out the board/CPU for an Intel i7-8700 and a Z370 motherboard.

I didn't go with FreeNAS as my previous post suggested I might, because ultimately I wanted complete control, so it's a plain Ubuntu Server machine that is fully managed by Ansible playbooks. In retrospect it was a mistake to try and delegate server tasks to an appliance like the Synology, and it was a further mistake to try and deal with that by getting the NUC - I should have just cut my losses and gone straight to a Linux server. Lesson learned!

Instead of getting lost in the weeds of purchase choices and justifications, instead let's look at some of the things I'm doing to the server with Ansible.

First up is root disk encryption - it's nice to know that your data is private when at rest, but a headless machine in a cupboard is not a fun place to be typing a password on boot. Fortunately I have two ways round this - firstly, a KVM (a Lantronix Spider) and secondly, one can add dropbear to an initramfs so you can ssh into the initramfs to enter the password.

Here's the playbook tasks that put dropbear into the initramfs:

- name: Install dropbear-initramfs
  apt:
    name: dropbear-initramfs
    state: present

- name: Install busybox-static
  apt:
    name: busybox-static
    state: present

# This is necessary because of https://bugs.launchpad.net/ubuntu/+source/busybox/+bug/1651818
- name: Add initramfs hook to fix cryptroot-unlock
  copy:
    dest: /etc/initramfs-tools/hooks/zz-busybox-initramfs-fix
    src: dropbear-initramfs/zz-busybox-initramfs-fix
    mode: 0744
    owner: root
    group: root
  notify: update initramfs

- name: Configure dropbear-initramfs
  lineinfile:
    path: /etc/dropbear-initramfs/config
    regexp: 'DROPBEAR_OPTIONS'
    line: 'DROPBEAR_OPTIONS="-p 31337 -s -j -k -I 60"'
  notify: update initramfs

- name: Add dropbear authorized_keys
  copy:
    dest: /etc/dropbear-initramfs/authorized_keys
    src: dropbear-initramfs/dropbear-authorized_keys
    mode: 0600
    owner: root
    group: root
  notify: update initramfs

# The format of the ip= kernel parameter is: <client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>
# It comes from https://git.kernel.org/pub/scm/libs/klibc/klibc.git/tree/usr/kinit/ipconfig/README.ipconfig?id=HEAD
- name: Configure boot IP and consoleblanking
  lineinfile:
    path: /etc/default/grub
    regexp: 'GRUB_CMDLINE_LINUX_DEFAULT'
    line: 'GRUB_CMDLINE_LINUX_DEFAULT="ip=10.0.88.11::10.0.88.1:255.255.255.0:gnubert:enp0s31f6:none loglevel=7 consoleblank=0"'
  notify: update grub

While this does rely on some external files, the important one is zz-busybox-initramfs-fix which works around a bug in the busybox build that Ubuntu is currently using. Rather than paste the whole script here, you can see it here.

The last task in the playbook configures Linux to boot with a particular networking config on a particular NIC, so you can ssh in. Once you're in, just run cryptsetup-unlock and your encrypted root is unlocked!

Another interesting thing I'm doing, is using Borg for some backups. It's a pretty clever backup system, and it works over SSH, so I use the following Ansible task to allow a particular SSH key to log in to the server as root, in a way that forces it to use Borg:

- name: Deploy ssh borg access
  authorized_key:
    user: root
    state: present
    key_options: 'command="/usr/bin/borg serve --restrict-to-path /srv/tank/backups/borg",restrict'
    key: "ssh-rsa BLAHBLAH cmsj@foo"

Now on client machines I can run borg create --exclude-caches --compression=zlib -v -p -s ssh://gnuborg:22/srv/tank/backups/borg/foo/backups.borg::cmsj-{utcnow} $HOME and because gnuborg is defined in ~/.ssh/config it will use all the right ssh options (username, hostname and the SSH key created for this purpose):

Host gnuborg
  User root
  Hostname gnubert.local
  IdentityFile ~/.ssh/id_rsa_herborg

Homebridge server monitoring

Homebridge is a great way to expose arbitrary devices to Apple's HomeKit platform. It has helped bridge the Google Nest and Netgear Arlo devices I have in my home, into my iOS devices, since neither of those manufacturers appear to be interested in becoming officially HomeKit compatible.

London has been having a little bit of a heatwave recently and it got me thinking about the Linux server I have running in a closet under the stairs - it has pretty poor airflow available to it, and I didn't know how hot its CPU was getting.

So, by the power of JavaScript, Homebridge and Linux's /sys filesystem, I was able to quickly whip up a plugin for Homebridge that will read an entry from Linux's temperature monitoring interface, and present it to HomeKit. In theory I could use it for sending notifications, but in practice I'm doing that via Grafana - the purpose of getting the information in HomeKit is so I can ask Siri what the server's temperature is.

The configuration is very simple, allowing you to configure one temperature sensor per instance of the plugin (but you could define multiple instances in your Homebridge config.json):

{
    "accessory": "LinuxTemperature",
    "name": "gnubert",
    "sensor_path": "/sys/bus/platform/devices/coretemp.0/hwmon/hwmon0/temp1_input",
    "divisor": 1000
}

(gnubert is the hostname of my server).

Below is a screenshot showing the server's CPU temperature mingling with all of the Nest and Arlo items :)

Screenshot


A little bit of automation of the Trello Mac App

Trello have a Mac app, which I use for work and it struck me this morning that several recurring calendar events I have, which exist to remind me to review a particular board, would be much more pleasant if they contained a link that would open the board directly.

That would be easy if I used the Trello website, but I quite like the app (even though it's really just a browser pretending to be an app), so I went spelunking.

To cut a long story short, the Trello Mac app registers itself as a handler for trello:// URLs, so if you take any trello.com board URL and replace the https:// part with trello:// you can use it as a link in your calendar (or anywhere else) and it will open the board in the app.