The existing Linux-based firewall for QubesOS uses a lot of memory, which is a shared resource among all virtual machines and thus limited. The firewall based on the minimal operating system MirageOS uses less memory, and has a smaller attack surface.
The first milestone was the development of a state machine to classify network packets. This step also included a good test coverage and the use of fuzz testing via the crowbar library, which combines AFL with property based testing in OCaml. All these building blocks were combined into a unikernel which can read rule changes from QubesOS via the Qubes database. Also, we integrted a DNS client, to make it possible to write rules in a human readable form containing domain names, and not just as machine readable IP addresses.
The user group of this project consists of activists, journalists, security professionals and other users of the Qubes operating system. Using the lightweight firewall, they can work better since it uses less memory, and they can conveniently change filter rules without having to rebuild the firewall like in the past. In the past, some users were forced to switch off the firewall due to memory restrictions.
The firewall is not available as a template vm in Qubes yet, although this is a longterm goal. Currently, there is an obstacle: Qubes uses a new virtualization scheme in its latest version, other than MirageOS (pv vs pvh). MirageOS cannot handle this yet, and the user has to switch the virualization mode manually to the old mode until MirageOS supports the new mode. It is already possible to download the firewall release from GitHub or to build it manually from source. Of course it would be most convenient to select it from the Qube template dropdown directly.
All Qubes users can use the lightweight firewall and save memory. It is written in OCaml, a functional programming language with advantages regarding secure programming (e.g. memory safety, immutability by default). It was announcend on the qubes users mailing list and has a group of existing users already.
]]>Install acmetool as explained on https://github.com/hlandau/acme. To set up your account, you can run sudo acmetool quickstart It asks you whether you want a test certificate or the real thing. A test certificate is not from a trusted CA, but it has less restrictive rate limiting, and is good for testing the automated certificate retrieval process. To get a certificate, you need to prove that you control your webserver. To do this, you need to "solve a challenge". I picked the stateless challenge, which just requires a change to your webserver's setup. Acmetool quickstart will ask you about what challenge you want to pick.
For the stateless challenge, you need to add a section to your webserver config that handles challenge traffic on port 80. For nginx it looks like this:
server {
listen 80 default_server;
listen [::]:80 default_server;
..
location ~ "^/\.well-known/acme-challenge/([-_a-zA-Z0-9]+)$" {
default_type text/plain;
return 200 "$1.ACCOUNT_THUMBPRINT";
}
..
Your account thumbprint is different for the staging / test certificates and for the real certificates. You can get both of them via
sudo acmetool account-thumbprint
There is a lot more information about the challenges at https://hlandau.github.io/acme/userguide#web-server-configuration-challenges.
Now that you've set up the challenges, you are ready to get some certificates for your domains. You can do this with
sudo acmetool want mydomain.net
If you have multiple subdomains and want to use the same certificate, you can request them all in one want
command. If you want to use multiple certificates, you should issue multiple want commands.
To make your webserver use the certificate, you have to add two more bits of configuration, ssl_certificate
and ssl_certificate_key
.
server {
listen 443 ssl;
..
server_name mydomain.net;
ssl_certificate /var/lib/acme/live/mydomain.net/fullchain;
ssl_certificate_key /var/lib/acme/live/mydomain.net/privkey;
..
Restart your webserver (for me: sudo /etc/init.d/nginx restart
), and you're done.
I program in many languages and each one has different ways to build projects. Since it is hard to remember the build specifics, I put them in a Makefile. Even when I wrote my thesis in LaTeX, I had a Makefile.
For data analysis projects I also used Makefiles, because some steps take time and it does not always make sense to rerun the entire pipeline, but it is important to run for example the cleanup step before we can run the normalization and the statistical analysis.
That's three main advantages:
At the same time, I had a website that was runnig wordpress. And that wordpress blog took part in a DDoS pingback flood attack - very embarrassing. At that point I just used wget -r to download the entire wordpress blog, and serve it as a static site. :-D Awesome hack, but quite hard to update the blog content after that.
]]>At the first training hike in the rain with a twitter friend from Berlin, @Mirabellensaft, we managed to hike 14 km, and then her back did not want to walk any further. She wrote to me after a trip to the doctor that she could not participate in the hike. I felt very sorry, because she and @Mamsellchaos were the two people that gave me this crazy idea in the first place.
I was hooked now, I really like hiking, and despite the rain the countryside was really nice! On the internet I found a training hike by Carola and her group "earnyourbacon". To the rescue. :D And they were already in full training mode. Phew! I joined their next hike and it was awesome to climb Teufelsberg with this giant group and explore the surroundings of Berlin. The group was so big that all speeds of hiking were possible. 45 km was really a challenge though. In the end I walked as slow as a tortoise, because "the countryside was so beautiful" and my pants caused a rash.
Photo: earnyourbacon taken by Carola Kessler.
I started to talk to friends and colleagues about the hike. Three people said right away: You're gonna do it. <3 The best mentor that I ever had (+ friend of the great outdoors from Colorado), a teammate, and my partner.
At the following training hikes, we graduated to 50 and 55 km distance and I started experimenting with music, audiobooks and conversations vs silence and got to know many new people. I also had some hikes by myself and after the 50 km I did 20 + 20 on the following days during Easter in the Uckermark. The long hike of 60 km I missed because of a business trip to Brooklyn. Just 2 weeks to get rid of the jet lag until the hike. I was enrolled in start group four with team "earnyourbacon" and Mirabellensaft was in group 8 with team "Humpelstilzchen". Hats off to this much courage!
Then there was a forecast with rain. My rain jacket is a bike jacket, no hood, because there's supposed to be a helmet. On the evening before the hike, I quickly sewed a hood out of an old jacket. On the day of the hike I tried to sleep as long as possible. Then I made a favourite dish. Omurice - Japanese rice omelette. Then I was late and missed the train, but I also was one hour off. Oh my god, 45 mins before the start I arrived in Erkner, and not 2 hours early, as planned. Hupp hupp, hand in the food bags, and look for the group. Just a picture and off we go!
The start music is way too cheesy for me, I have to giggle. I meet a few ladies from Switzerland. International guests! Then I talk to two climbers from the earnyourbacon start group about climbing, mental preparation for the hike (for me the book "100km to the next beer"), and Canada, and am waiting desperately for the toilets at Mueggelsee. And there we are, at Strandbad Mueggelsee, the first food reststop. I'm not really hungry, because I ate a Clif bar and before we started I had eaten my first piece of omelette. It's a bit chaotic and very windy, but I find Miri and Andreas and their gang on a bench and try to put my Iso drink powder into my bottle, without it flying away in the wind. A banana and a soft roll, and onwards. I am happy to se Carola and Carsten, but lose them immediately in the crowd.
On a dusty path we walk into the sunset, at the side of an S-Bahn track. It's getting less crowded, and I overtake and get overtaken, but it's not as full of people as at the start. In front of me is someone with a "Hello Fuckface" adbusted patch from The North Face on his bag. Great, I faked this myself as "Das Nord-Gesicht" when I was younger. The way's changing, sometimes asphalt, sometimes dust. I'm walking behind the young nerd with long hair and trekking poles, I know him from our training team, he has a good hiking pace.
We walk through an S-Bahn station, on the other side there are giant stairs and many people sit on them and are take a break. Kilometer 33? I sit down, take off my lucky rainbow socks, wash the dust off my feet, put some foot balm on, and put on clean socks. Like new! I see other people taking care of their feet, spray-on skin, tape. Luckily this is not necessary for my feet yet, Hirschtalg footbalm is my only drug. :-D Off we go. And again, dust path. Well, that was worthwhile, washing my feet. And then we make a right turn into the forest and it's getting dark. I put on my headlamp.
Scene change. Or that's what it feels like. People are lounging at the side of the path, and it's not always clear if they are taking a break or have given up. It's dark, and everyone gets slow and walks as one long line of people on the narrow forest path. Everyone orients themselve based on the preceding person, the gaps between the trees are narrow. But the path is a good old well-trodden dirt path. Again and again there are traffic jams in this long chain, for example at the transit into an underpass next to a river. The first people start to grumble that it is too narrow or too slow for them. I put on my headphones and listen to the song "Easy" by Erobique. I always have to laugh listening to this song. :-D
The snake goes on and winds itself around corners and across glades. I have a lot of fun with this light-snake, really cool if you look behind you, and start listening to Onionbrain - Kenophobia (creepy psytrance). It is a bit freaky, because the cone of light creates a kind of tunnel vision. My parner said "you'll see ghosts". We get out of the forest, and again people are sitting at the roadside. Sometimes they are laying down. One time I get really scared when a person is laying on their side and not moving. This does not look good. But there is somebody with them and is helping them. I search for eye contact and decide that it is okay to move on, because the other person looks really hopeful and nice. But this is not an easy decision. Phew.
After a really beautiful track in the moonlight on a wide village road while I am listening to Tibetan throat singing (really great, I tell you), I notice that it is really cold. And I am in shorts. I have no interest in putting on my long socks now, the hiking keeps me warm. So I eat clif bars and iso drinks constantly. And then all of a sudden a long-legged guy with poles and backlight is in front of me, who has exactly my pace! Wow! I settle behind him and we have an amazing run, without him knowing that I am there. We overtake hundreds of people, at least that's what it feels like, it feels like dancing! I loop through my Kumbia-Electro mices from Nicola Cruz and Luzmila Carpio, so this takes a few hours actually. Very nice night music for this forest.
And just as I ate my last Clif bar: the second food reststop. With the super-strider we queue at the first line, because we think it has the food bags. Instead, the guy asks us "so you want to give up?" and my predecessor replies "Oh god no, never! We wanted our food." and we have to laugh. Somehow this is more party-like, Techno, Picknick tables and benches, getting the food bags. Making new iso drink and eating my second Omurice. Icy temperature, 6 degrees celsius? On to the porta potty. On the way back I meet Melanie, say hello, but glare at her with my headlight, oops! Sorry again. Altogether a strange feeling at the food stop, more like a party or a fairground than a care taking station. People shouting "she should go there with her caaaard if she wants to give up" and other bizarre scenes.
I am on the road again and meet a new companion right away. We talk for a long time, the hiker from Berlin and I, he started hiking with the shop "Sport Scheck", and survives even though he has a busted knee. And then he tells me how his wife got pregnant for the first time, they were super happy for four weeks, and then she got a cancer diagnosis. He tells me about their struggle, how his wife fought the cancer, but lost the baby, while all couples around them were having babies, and how they went through all of this together. A story that really moved me and came close to my heart, and a really awesome conversation partner. We walk together for a long time, until I get hit by a wave of tiredness, need some music to dream a bit and we say goodbye to silently walk together some more.
And suddenly there is a guy on the horizon, at the side of the road. When I reach him, I ask "do you know the way?". He says "that's why I'm here. Everything okay? Go in here and behind that glowing Lamp, go on the sports court. Ok, great, I take that corner and walk straight to some portapotties. And then I ask people if this is the Foodstop Rehberge, because not a single person is eating. Everyone sits there, looking discontented. I was directed to get my food bag. Then I sit down, wash my feet. And put my food in my backpack. Behind me, a group of Swedes is cracking jokes at a picknick table. I'm not really hungry yet. But now I got my breakfast, including a thermos of coffee! Nothing can stop me now. Fresh foot balm, clean socks, and putting the shoes back on. I am walking towards the water refill stations. On my way, I meet a big group of "earnyourbacon" friends who are also sitting and looking lost. I ask them how they are and whether they plan to continue. They say they want to quit right now. I try to encourage them and say "this is probably just the night, it's cold right now, are you sure?" Then they say the hike is officially cancelled. I am completely surprised, can't really understand, I was just getting ready to continue. I ask where the hike continues, and somebody points me to the water refill station.
Confused, I walk over to the drinking water, refill my iso drink and start thinking. On top of that, it also starts raining. Good thing there is a tent here. At the water refill, there's a helper that I already met during my training hikes. I ask her if we really cannot continue. She says, "once in a while, people still continue the hike". Then our wonderful wanderer with the magic pace appears. I talk to him, and ask him if he wants to continue hiking. For 40 more kilometers, and it's not clear if there will still be food and water. He says, this is too risky in his opinion. The helper says "the girl over there with the ponytail wants to continue". I talk to her, but she wants to wait for her friends and decide with them. I have no plans for this weekend. I am feeling fine. But walking alone is risky without any infrastructure. I have a rescue blanket, a hat, gloves,a lamp that works for 140 hours, and phone batteries for three days. And 20 years of outdoor experience. I have enough food, and am able to carry upt to 3l of water and have and half a liter of coffe and half a liter of milk. And I am feeling fine. Difficult. One second I wrangle with the decision and say "Phew, I guess I really go home then, if no one continues." I really did say this sentence. But did not do that.
Because then, a small, feisty guy came by and said "I continue, out of spite!". :-D I ask him if we want to go together. There is a dramatic scene because my new cowalker reacts to a request from a person at the side "do we need an ambulance?" by yelling "ambulance needed, over here!". To be helpful. But the people laying down get angry and say "not really, he just woke up again". People are a bit on the edge. We say something calming, as we intended to be helpful, and walk on.
We start walking, orient ourselves, it is already beginning to dawn, I think. Soon it is clear that I walk faster than my new comrade, and he's not sure for how long he actually wants to continue. He phones his friend to talk about the situation. On the horizon I see two boys. My comrade wants to be slower, but is happy with me walking towards the others, he still has his friend on the phone who will pick him up. So I hike towards the two people on the horizon.
With them I have a really great time. They left the third person of the group behind and intend to go all the way to the finish line. A bearded and a non-bearded guy. :-D We talk about the entire situation and wonder how many hikers are still continuing right now. On our way we then meet Michael and Gabi. Both are experienced Marathon runners, one age group above ours. Cool. Come with us. Together we made our way through dawn over different types of difficult terrain. To motivate the others, I offered my coffee, and one of the guys said, he'd already drank espresso in the night from someone's thermos. Michael turns around and says "that was me!". :-D At a bigger road, we met a van with many resting folks. If I understood correctly, this was the hiking group of Gabi and they were stressed and starting a loud quarrel. Michael and I just went past them as we had no energy for fights. We almost went past the wald elf Julia in her camouflage poncho, who was waiting to get picked up by car. She blended in with the woods perfectly. I actually yelled out in surprise "Wow, there is another person!" because she was camouflaged so well in the vegetation.
At some point we walked on a really nice forest path and on the left side there was a bridge. We hiked past it, and there was another, lightning-fast and super fit looking female hiker, and she explained we have to cross that bridge. And there it was! The last food rest stop, Buckow. It was still there!
Together with the female hiker we arrived at the checkpoint and were welcomed in the most awesome way, several people clapped and led us immediately into the house, towards chairs and brought us food and water. Thousand thanks to Adam and the others from this stop in Buckow, you really revived us again! Gabi and I both had Muesli in our bag and had a luxurious breakfast, I drank a cup of coffee, went to a real restroom, it was like a dream. Together with the fit female hiker we thought about another sock change, she even wanted to give me some new socks and also a tomato, but I decided against another change. Even though I had some small stones in my shoes, I thought it'd be better not to look inside, but rather tough it out for the last few (haha) kilometers.
And then we continued. The guys had already left, but somehow we overtook them. The last 20 km were tough. I knew that at km 85 there will be the bonking point, like people tell it also about marathons. But at that point it actually was kind of okay still. Afterwards it got worse. In addition Gabi kept returning to the quarrel with her running group, which we kept meeting again and again (or their van). For me that's difficult to understand but I wasn't involved. Everyone is fighting against themselves already, and never against others. I don't intend to do that. Everyone I met on the road helped me in one way or another, and I carried them all to the finish line in my heart. :-D
We met an old man who walked from the other end to meet his grandson who was hiking the 100. He had cold drinks and was very encouraging. We dragged ourselves along the path. At km 90 Gabi's husband arrived by bike, and took our bags and talked to us. Gabi had fallen and I suggested to take a really short break, for safety reasons, and there he was. The super fit female hiker already got a lift by car from Neuhardenberg. The last km were really just asphalt, next to a highway, and every kilometer was a fight. At the sign of the village Gusow we took some pictures, I had really itchy legs and could not stand still. Further on over the tough hard asphalt. Sometimes Gabi pulled me along on her scarf, me behind her, and I had encouraging words, listened to the "Easy" song on repeat without avail, and Gabi's husband had encouraging words for both of us. :-D It was not easy, no matter how often I listened to the song. Then I also led us astray for half a kilometer because I could not navigate any more and my GPS was just the route but no maps. Luckily, after one turn we were back on track. I was swearing and muttering, but somehow this expression of pain was helpful. :-D
Quality phone pic in Gusow. :-D
And then the last turn. Phew, another time a straight long path until the horizon. Limp, limp. But then. The station! We made it. Gabi was somewhere behind me, and I did not realize much any more, I really had to sit down. Gabi's husband gave me the backpack and I thanked him with the words "you saved my day" :-D He really did. I hardly said goodbye to Gabi, if you read this I send you my greetings and give you a hug! I hope I remember all the names correctly from that night.
At the station's waiting hut, in which I was sitting, I sat next to Klaus. The station Gusow was a sad sight, folded benches, and people next to a car, who clapped once in a while, but somehow not for me, or did I not realize it? A bit sad for a finish line, but the real finishing happens in the mind anyways. It was 1:45, I sent a few messages and got a nice congratulatory call from a friend, who had followed along my km-tweets during the night on his phone. He said I was way too early! :-D
Klaus on that bench was also the bomb and said he might as well stand, in case I wanted to put my legs up on the bench. And this after he himself had hiked the entire distance. I immediately said no, and layed down on the street and put my feet on one seat, next to where he was sitting. And in this way we had a good conversation, and he notified me in time to limp slowly to the train station platform. And he took care of the ticket situation and also of planning my transfers to home, I was really not that lively any more. As an experienced hiker, he had a few 100km hikes under his belt and knew exactly how to be helpful. Together with a handful of finishers we did the Zombiewalk through Lichtenberg station and Friedrichsstrasse and he, being an old hand, gave me a good overwiew about the long distance hiking scene. I stumbled in slow motion out of the train, oh no, wrong exit, so far to home, and then to the 5th floor, no elevator. Fell asleep, woke up at 5 am, and made soup. My body is only slowly coming down from this level of energy burning. If I eat a sandwich, my body wants five of them. :-D
]]>I was thinking about this for a while, and also thought I don't want to use an external service such as disqus - why depend on another service if I can learn to build something instead? Then I thought a bit more -- how about using another provider I depend on already? How about using github for my comments?
So I set out to build a comment service that creates github pull requests to reflect peoples comments as changes to the source of my posts. The gitub open pull request queue serves as my comment moderation queue. I either close a pull request and the comment does not pass moderation, or I merge the pull request, and the site gets rebuilt with the comment appended to it. Due to the parasitic nature of this service, I called it "Zeckernews" - Zecke means tick, the blood sucking insect, in German.
We need a few dynamic pieces for this, but they can be mostly on the client. All the form actions and headers drawn and other fun stuff are in JavaScript on the client. We only need two tiny and well-defined servers: A comment server and a deploy server. The comment server accepts the requests from the comment form, and creates github pull requests for each one. We can use github's API for this. First we create the local change and push it:
function composePR(formData, callback) {
var commentfile = options.local_repo + "/content/"
+ formData.file + "/comment" + formData.nonce + ".md";
var pushurl = "https://" + options.token
+ "@github.com/linse/zeckernews.git " + formData.nonce
// git branch
exec("cd " + options.local_repo + " && git checkout -b " + formData.nonce
// change source file
+ " && echo \"\n\n\n____\n\n**" + formData.name + "** posted a message:\n\n> "
+ formData.message + "\n\n\" >> " + commentfile
+ " && git add " + commentfile
// git commit
+ " && git commit -m \"" + formData.name + ": " + formData.message + "\" content"
// git push fails if branch exists - thats why we add a nonce / hash
+ " && git push " + pushurl + " ; git checkout master", callback);
}
Then we format it as an open pull request for github's API:
function postPR(formData) {
var data = '{'title': 'Comment PR from ' + formData.name,
'body': 'from the form',
'head': "' + formData.nonce + '",
'base': 'master'}';
console.log('postPR');
console.log(data);
sendGithubRequest('POST', '/repos/linse/zeckernews/pulls', data);
}
In the last line we see how we send it via POST to github. This function looks like a standard HTTPS request to github's API server:
function sendGithubRequest(method, path, content) {
var post_req = https.request({
method: method,
hostname: 'api.github.com',
path: path,
headers: { 'Authorization': 'token '+ options.token,
'User-Agent': 'zeckernews' },
},function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log('Response: ' + chunk);
});
});
// post the data
post_req.write(content);
post_req.end();
}
Then I wrapped the whole thing in a tiny https server and run it as a daemon. Let's try this out and use the comment form at the bottom of a post. So this is the first step. We made a comment and got an open pull request on github. What can we do with this?
The next step is comment moderation. I have to take action on the pull request queue. I either close a pull request. Or I approve it and merge the code change that it introduces.
Both actions trigger my deploy server, via github's webhook interface. This is the second server I have running. The deploy server is even more concise than the comment server.
var server = https.createServer(options, function (req, res) {
var body = "";
req.on('data', function (chunk) {
body += chunk;
});
req.on('end', function () {
// webhook is called
pullReq = JSON.parse(body);
if (pullReq.action == 'closed') {
pull();
removeBranch(pullReq.pull_request.head.ref);
// when pr was closed by merge
if (pullReq.pull_request.merged_at != null) {
rebuildZeckernews(puts);
}
}
res.end("Send me moar pull requests!");
});
})
// Listen on given port, IP defaults to 127.0.0.1
server.listen(options.port);
You can see in the code that the case distinction is a bit convoluted. We know that /something/ happened to a pull request if the webhook is triggered. We would like to know whether the action was closing the PR - and then we still need to know whether it was closed by merge or just closed and abandoned. This we can learn by parsing the result JSON and looking at the action and merged_at fields. Based on these cases, we rebuild the blog post page if necessary.
]]>In March 2018, I attended my first MirageOS hack retreat in Morrocco. MirageOS is a library operating system which allows everyone to build very small, specialized operating system kernels that are intended to run directly on the virtualization layer. The application code itself is the guest operating system kernel, and can be deployed at scale without the need for an extra containerization step in between. It is written in OCaml and each kernel is built only with exactly the code that is necessary for the particular application. A pretty different approach from traditional operating systems. Linux feels huge all of a sudden.
I flew in from New York via Casablanca to Marrakesh, and then took a cab to the city center, to the main square, Jemaa El Fnaa. At Cafe de France, Hannes was picking me up and we walked back through the labyrinth of the Medina to the hostel Riad "Priscilla" where we lived with about 20 MirageOS folks, two turtles and a dog. We ate some food, and there were talks about Mirage's quickcheck-style fuzzing library Crowbar, and an API realized on top of a message queue written in OCaml.
Coming from compiler construction in Haskell and building "stateless" services for information retrieval in Scala, I have a good grasp of functional programming. The funny problem is I don't know much about OCaml yet.
At Etsy, I was part of the Core Platform team where we first used hhvm (Facebook's hip-hop virtual machine) on the API cluster, and then advocated to use their gradually typed "hack" language to introduce typing to the gigantic PHP codebase at Etsy. Dan Miller and I added types to the codebase with Facebook's hackificator
, but then PHP 7 added the possibility of type annotations and great speedups, and PHP's own static analyzer phan
was developed by Rasmus Lerdorf and Andrew Morrison to work with PHP's types. We abandoned the hackification approach. Why is this interesting? These were my first encounters with OCaml! The hack typechecker is written in OCaml, and Dan and I have read it to understand the gradual typing approach. Also, we played with pfff
, a tool written in OCaml that allows structural edits on PHP programs, based on the abstact syntax tree. I made a list to translate between Haskell and OCaml syntax, and later Maggie Zhou and I used pfff
to unify the syntax of several hundred endpoints in Etsy's internal API.
At the MirageOS retreat, I started my week reading "Real World OCaml", but got stuck because the examples did not work with the buildsystem used in the book. Stephen helped me to find a workaround, I made a PR to the book but it was closed since it is a temporary problem. Also, I started reading about OCaml's "lwt" library for concurrent programming. The abbreviation stands for lightweight threads and the library provides a monadic way to do multithreading, really similar to twitter futures in Scala. Asynchronous calls can be made in a thread, which then returns at some point when the call was successful or failed. We can do operations "inside" lwt with bind (>>=
) in the same way we can flatMap over Futures in scala. The library also provides ways to run multiple threads in sequence or in parallel, and to block and wait. In the evening, there was a talk about a high-end smart card that based on a private start value can provide a succession of keys. The hardware is interesting, being the size of a credit card it has a small keypad and a screen. Some banks use these cards already (for their TAN system?), and we all got a sample card to play with.
One day I went swimming with Lix and Reynir, which was quite the adventure since the swimming pool was closed and we were not sure what to do. We still made it to the part that was still open, swam a lot and then got a cake for Hannes birthday which lead to a cake overflow since there were multiple cakes and an awesome party with candles, food and live music already. :D Thanks everyone for organizing!! Happy birthday Hannes!
I started reading another book, "OCaml from the very beginning", and working through it with Kugg. This book was more focused on algorithms and the language itself than on tooling and libraries, and the exercises were really fun to solve. Fire up OCaml's REPL utop
and go! :D
At the same time I started reading the code for solo5 to get an understanding of the underlying hypervisor abstraction layer and the backends we compile to. This code is really a pleasure to read. It is called solo5 because of MirageOS's system calls, initially a set of 5 calls to the hypervisor, called hypercalls which sounds really futuristic. :D
So that's the other fun problem: I don't know too much about kernel programming yet. I did the Eudyptula (Linux kernel) challenge, an email-based challenge that sends you programming quests to learn about kernel programming. Over the course of the challenge, I've made my own Linux kernel module that says "Hello world!" but I have not built anything serious yet.
The next things I learned were configuring and compiling a MirageOS unikernel. Hannes showed me how this works. The config system is powerful and can be tailored to the unikernel we are about to build, via a config file. After configuring the build, we can build the kernel for a target backend of our choice. I started out with compiling to Unix, which means all network calls go through unix pipes and the unikernel runs as a simple unix binary in my host system, which is really useful for testing.
The next way to run MirageOS that I tried was running it in ukvm. For this setup you have to change the networking approach so that you can talk from the host system to you unikernel inside ukvm. In Linux you can use the Tun/Tap loopback interface for networking to wire up this connection.
We had a session with Jeremie about our vision for MirageOS which was super fun, and very interesting because people have all kinds of different backgrounds but the goals are still very aligned.
Another thing I learned was how to look at network traffic with wireshark. Sidney and I had previously recorded a TLS handshake with tcpdump and looked at the binary data in the pcap file with "hexfiend" next to Wikipedia to decode what we saw. Derpeter gave me a nice introduction about how to do this with wireshark, which knows about most protocols already and will do the decoding of the fields for us. We talked about all layers of the usual stack, other kinds of internet protocols, the iptables flow, and bgp / peeringDB. Quite interesting and I feel I have a pretty good foundational understanding about how the internet actually works now.
During the last days I wanted to write a unikernel that does something new, and I thought about monitoring, as there is no monitoring for MirageOS yet. I set up a grafana on my computer and sent some simple data packets to grafana from a unikernel, producing little peaks in a test graph. Reynir and I played with this a bit and restructured the program.
After this, the week was over, I walked back to Jemaa el Fnaa with Jeremie, I feel I learned a ton and yet am still at the very beginning, excited what to build next. On the way back I got stuck in a weird hotel in Casablanca due to the flight being cancelled, where I bumped into a Moroccan wedding and met some awesome travelling women from Brazil and the US who also got stuck. All in all a fun adventure!
For my JavaScript program, the fractal is an equation that defines lots of "circular" functions, one for each parameter. So for each step of our iteration, we get a closed path that resembles a circle, and defines whether a point in 2d is inside or outside of that function - or on the island vs in the water. A constant is added in each step, making the resulting shape more and more precise. For us this is happening in 2D.
// a and b are our components to start with,
// x and y is what we get by adding a and b to it
function iter(a, b, x, y, ba, mi) {
var n = 0;
var x2 = x*x;
var y2 = y*y;
do { // z -> z^2 + c
y = 2*x*y + b;
x = x2 - y2 + a;
x2 = x*x;
y2 = y*y;
n++;
}
while (((x2+y2) < ba) && (n < mi));
return n;
}
If you go to the research page header and play with the fractal explorer, and look just at the first preset (a view of Mandelbrot set) and do only one iteration and redraw, the screen is filled with one color. If you do two iterations you see a shape emerging. It is just a circle-like bubble. With the third iteration you see a T-like bubble inside of that one, with a third color, and so on and so on. We see the fractal being constructed and getting more fine-grained. Let's see what it looks like. This image is for four iterations. Six iterations. 20 iterations - this looks like the Mandelbrot set already. But if we zoom in we see it is very coarse, still.
Determining how every pixel should be colored is a lot of work, we have to check every pixel via the equation for each 2D coordinate and iteration value. To speed this up, we can use JavaScript webworkers. Each worker computes one row of pixels in parallel using multithreading. You can try out the computation without or with multithreading (checkbox) and see the speedup we can get.
]]>