Goran Mekić

Neural and Impulse Blending

Tue, 15 Jul 2025 00:35:00 +0000

The idea of blending is to make two amplifiers and mix them into a single sound. The reason people do that is because different amplifiers have slightly different sound which combined together sounds better than any single one of them. It’s the same like having a single voice vs choir. With sounds it is simple: you just mix them. With neural amp modeler, there are few more steps. First you have to capture the sweep. Then you have to mix the captures into a single track. To do it properly, you need to normalize the captures and lower the track’s volume to -5dB, because when two tracks combine the resulting volume will be bigger. If you mix more than two tracks, you have to lower their volumes even more. Once you mix the captures into a single clip, normalize it. After that all you have to do is create NAM profile out of that clip.

Mixing impulse responses is even easier. You create the impulses, mix them into a single clip and that’s it. You have to watch out for the volume, that it doesn’t clip and normalize it in the end. Enjoy your sound!

Impulse Response Sweep

Mon, 12 May 2025 08:55:00 +0000

There is another way to capture IR, and it is using sweep frequency sound. That means that the file containing the sweep will have 20Hz to 20kHz frequencies, starting with low and increasing the frequency. The idea is to capture how the system, in this case power amp, guitar cabinet and microphone, respond to different frequencies and create IR based on that. For that purpose you can use Carla and LSP Profiler. I like to use mono version, but the stereo version is available if you need it. You need to start Carla and add LSP Profiler as a plugin. In patch bay of Carla connect it like the following picture.

In program like QJackCTL connect Carla’s IO to the hardware IO. As for the hardware setup and connections, consult the previous post about impulse resposes. First we need to calibrate the LSP Profiler. That means setting the right sound levels and latency.

Turn the volume down on your power amp before you enable Calibrator, as it sends loud signal, then enable it and turn the volume gradually up until you get around -3dB. When satisfied with the levels, disable it. Click Measure in the Latency Detector. It will send pulses of sound to figure out what is the round trip time for the sound. In the Test Signal, increase Coarse Tuning to be as long as possible. When all is set, press Profile button. It will start with a pulse, then few seconds of silence, then sweep sound through 20-20000Hz range. Once the LSP Profiler captures the sound from the microphone, it will need up to few seconds to make an actual IR profile. When that’s done, press Save and create your .wav file.

I have to note that quality of such IRs is lower than those I did with Dirac pulse. As I am still new to this I assume I’m not tuning something right as these two techniques should produce the same result.

Impulse Response

Sun, 11 May 2025 10:56:00 +0000

DANGER!!! THE FOLLOWING PROCEDURE MIGHT CAUSE DAMAGE TO YOUR EQUIPMENT!!!

I have power amp that has limiter, short circuit protection and overheating protection, so I know my gear can handle the procedure, but please don’t assume that every combination of amp/cab is safe to be used! For example, I have no idea how tube amps react to this procedure and you might burn the tubes. Please be safe, first!

There is also a procedure using sweep frequencies, but I’ll describe that in another post. That procedure should be safe for any equipment.

Dirac Pulse

Impulse response is used to emulate acoustic characteristic of the system. In other words, it can emulate the sound of a guitar cabinet. It can also capture reverb of a place, like hallway or church, but I’ll just stick to guitar cabinet emulation in this post. The core of this technology is in the title as we will record how system, in our case power amp, cabinet and microphone, responds when we feed it just a single impulse. That pulse is also called “Dirac pulse”. In the wav file, it would be single sample with full amplitude and all samples before and after it are zero. It is said that such pulse contains all frequencies with equal amplitude, but the math behind it deserves a post on its own. Let’s first generate such .wav file in audacity. First we need a track to work on.

Tracks -> Add New -> Mono Track

Then we need some samples to work on. It’s easiest if we generate 1 second of silence and work with that.

Generate -> Silence

You’ll need to zoom in as much as possible on X and Y axis. Now raise first sample to the top and leave the rest on laying on zero. The result should look like this.

This clip actually has second sample raised and the reason I did that is because it is hard to see first sample because of the clip edge. Save the project and export as .wav.

Capturing Impulse Response

In Ardour we will need two mono tracks. One will have the Dirac pulse we just generated, second one will capture the output of the microphone directed at the guitar cabinet. With PC I will mark computer + audio interface, just to make it shorter.

PC (dirac) -> Power amp -> Cabinet -> Microphone -> PC (impulse response)

This means that dirac track should have its inputs disconnected and output connected so it is fed to power amp. The impulse response track should have all outputs disconnected and input connected only to the microphone.

There is a catch. To suppress pops and clicks that appear between two clips, Ardour will add a small fade in to the start of the clip and fade out to the end of it. This will “eat up” our Dirac pulse, so we have to disable that. Zoom as much as possible until the curve for fade in/out is visible. Right-click on the little square at the end of the curve and click on Deactivate. You have to do that on captures, too.

Do few test captures to check the levels and then start capturing impulse responses. You can export the capture and open it in Audacity for trimming start and end if needed. I also like to add very small fade out at the end. For aligning the start of the clip I use z shortcut to find the zero crossing in the clip and I delete the samples before that one. That way no delay is introduced by impulse response. Save that project and export the .wav file. That file is your IR! Congratulations, you just made your first IR!

Neural Amp Modeling

Tue, 15 Apr 2025 13:30:00 +0000

Re-amping

Let’s say you’re a guitar player and want to record your track of a song. Typically you would enter the studio with your guitar, amp and cabinet and record what’s needed. But let’s say you decide to record your guitar without any processing by pedals or amps, glue the good takes together and then process that track with pedals and amps. That process is called re-amping.

Now let’s say you want that guitar track processed by a plugin, not hardware. The process of recording would be almost the same, but the process of getting the plugin to sound like an amp is quite a challenge. Today if we want to emulate the whole rig, it is best if it’s done in two parts: NAM and IR. IR is much older, and it stands for Impulse Response. NAM stands for Neural Amp Modeler and it’s just a few years old. IR is used to capture the sonic characteristic of reverb and guitar cabinets, the most. It can do other things, but for now let’s just say that IR in our case is used to emulate guitar cabinets. NAM in our case is used to capture and emulate preamp. It can also capture full rig but you can’t use any effects that are time based, like reverb or delay. The reason why I like NAM for preamp and IR for cabinet is because that way I can use IR cabinets captured over past two decades. As NAM is relatively new, it still doesn’t have such a huge ecosystem.

Preamp and NAM

One of the great things that came out of AI hype is Neural Amp Modeler, or NAM for short. To capture NAM, you have to separate preamp from the rest of the gear. As stated before, you can capture NAM of your whole rig, but today I want to capture just the preamp. In my case it is Engl E570. It’s pretty rare so I’d like to be able to replace it with a NAM loader pedal (hardware pedal that loads NAM and emulates my preamp). If you’d like to profile preamp that is part of amp head or combo, you’ll need to use FX loop. To be precise, you’ll use Send on the FX loop as that is the preamp’s output. What is needed for that beside preamp is audio interface, computer and re-amp box. My choice of operating system is FreeBSD, but once you have JACK configured, the rest is the same as Linux. On the mentioned computer I will use JACK and Ardour to capture the sound of preamp. Computer is connected to audio interface via USB, and in my case it is Presonus AudioBox 1818VSL. It has line level outputs, which is almost exclusively the case on audio interfaces, so I can’t just plug it’s output to the preamp’s input. The impedance and signal level of line output do not match those of the guitar input on the preamp. To convert line to guitar level, I am using Radial Engineering EXTC-Stereo. It has inputs and outputs on one side, which are connected to the audio interface, and send and receive on the other side, which are connected to the preamp. To be precise, audio interface output is connected to EXTC, send of EXTC is connected to preamp input and preamp output is connected to audio interface (line in). If I call computer + audio interface just PC to make it short, this would be the “diagram” of connections

PC -> EXTC -> preamp -> PC

What we want is to send a signal from the PC, process it with the preamp and send it back to PC for analysis. To do that, create Ardour session at 48kHz and create two tracks with names “sweep” and “capture”. Both tracks need to be mono. Connect sweep’s JACK output to the output on the interface which is connected to reamp box. Connect capture’s JACK input to the input on the interface which is connected to the preamp. That way you can record the output of preamp in capture track and play some signal on the sweep track. Sweep is the term left from IR, and I will explain why it’s called that in the follow up post. For now all you need to know is that it’s not “normal” sound, it is a signal created so that AI can learn from it. To be precise, AI will learn from sweep and captured signal and it will produce information how to digitally transform the sound so it is the same as if we just did the reamp. Input on sweep track needs to be disconnected and output on capture track, too. That way we are minimizing causes of eventual problems. To get the sweep signal, go to tone3000 and download “sweep signal” from the page (downloaded file will be called T3K-sweep-v3.wav). Now all you need to do is arm your capture track and record the output of the preamp while the sweep signal is fed to the preamp’s input. Once that’s done, align start and end of sweep and captured clip. Also, set session start and end to the clip’s start and end. Normalize the captured clip. Now we need to export sweep and capture to separate, mono files at 48kHz and 24bits. To do that, go to

Session -> Export -> Stem Export

If you don’t already have a format for 48kHz (or session rate) at 24bits, create one. We need wav files that are in no way further processed. That means disable trimming and normalization. In the Time Span tab select session range and only that one. In the Channels tab, select sweep and capture track and disable Apply track/bus processing. Now export the track to wav files.

Go to tone3000’s capture and upload dry (sweep) and wet (capture) wav files. Follow the form and wait for AI to process all epochs (100 by default). Once that’s done, you’ll have NAM file published on Tone3000 platform. Congrats!

Tips

For better performance I suggest compiling jack from ports with SOSSO library enabled. It can dramatically reduce DSP usage in JACK/Ardour.

To test your NAM you can use neuralrack-lv2 to load it together with some IR. Nice thing is that tone3000 already has some IRs, so download some and test it in neuralrack-lv2.

While there is also a way to only upload the capture, it never worked for me. The reason is that I couldn’t make Ardour produce the exact same number of samples in the capture file, that the sweep file has. With stem export, tracks are exported to files with the exact same length. Briefly talking to Tone3000 support, they told me that input signal should be uploaded only if custom signal is used to train the AI. I would love to not waste resources by uploading both files, but I always get an error while producing the NAM if I upload only the capture.

Update

When creating Ardour session, be sure to select “Audio Time” not “Beat Time” for “Default time domain”. That way capture and sweep will have exact same number of samples, and you can just upload the capture. I didn’t find a way to convert existing session from “Beat” to “Audio” time.

Rust and ioctl

Tue, 18 Mar 2025 22:23:00 +0000

When I explore new programming language, I like to poke audio. Rust is a new language for me and I need a pet project to learn. As I’m also using FreeBSD, that means audio device configuration is done via ioctl. Here is the summary of how to handle ioctl in Rust. Create the project, add nix crate enabling ioctl feature.

$ cargo new oss
$ cd oss
$ cargo add --features ioctl nix

For now I will just exclude AudioInfo from the code and show you the rest. The following is an example of how to handle two ioctl calls in Rust. One writes integer and one reads structure. Ignore for now that logicaly, setting channels before getting information about hardware is wrong. I wanted to show i32 version first, struct later.

use nix::libc;
use std::fs;
use std::os::fd::AsRawFd;

const SNDCTL_DSP_MAGIC: u8 = b'P';
const SNDCTL_DSP_CHANNELS: u8 = 6;
nix::ioctl_readwrite!(oss_channels, SNDCTL_DSP_MAGIC, SNDCTL_DSP_CHANNELS, i32);


const SNDCTL_INFO_MAGIC: u8 = b'X';
const SNDCTL_ENGINEINFO: u8 = 12;
nix::ioctl_readwrite!(
    oss_audio_info,
    SNDCTL_INFO_MAGIC,
    SNDCTL_ENGINEINFO,
    AudioInfo
);

fn main() {
    let devpath = String::from("/dev/dsp");
    let dsp = fs::File::open(devpath).unwrap();
    let fd = dsp.as_raw_fd();
    let mut channels: i32 = 2;
    let mut audio_info = AudioInfo::new();
    unsafe {
        oss_channels(fd, &mut channels).expect("Failed to set number of channels");
        oss_audio_info(fd, &mut audio_info).expect("Failed to get info on device");
    }
    println!("channels = {}", audio_info.max_channels);
    println!("rate = {}", audio_info.max_rate);
}

Let’s just concentrate on the three lines after use block. How did I know what values to use? For start you have to look in /usr/src/sys/sys/soundcard.h (assuming that’s where your FreeBSD source tree is). Let’s take a look.

#define SOUND_PCM_WRITE_CHANNELS	_IOWR('P', 6, int)
#define SNDCTL_DSP_CHANNELS	SOUND_PCM_WRITE_CHANNELS

As the call is _IOWR I know I have to use ioctl_readwrite macro, and it’s obvious where P and 6 come from. As the third argument to _IOWR is int I know I have to use i32 as fourth argument to ioctl_readwrite. Also, the first argument to that macro is the function name that is generated and used later in the code.

For getting information about underlaying hardware, OSS uses ioctl with a struct. Let’s see how it’s defined in the FreeBSD source

#define SNDCTL_ENGINEINFO	_IOWR('X',12, oss_audioinfo)

From this I know I have to use ioctl_readwrite, X for MAGIC number and 12 for the argument. The last one is to figure out how to work with the struct oss_audioinfo. Here’s that structure.

typedef char oss_longname_t[64];
typedef char oss_label_t[16];
typedef char oss_devnode_t[32];

typedef struct oss_audioinfo
{
    int	 dev;
    char name[64];
    int  busy;
    int  pid;
    int  caps;
    int  iformats;
    int  oformats;
    int	 magic;
    char cmd[64];
    int  card_number;
    int  port_number;
    int  mixer_dev;
    int  legacy_device;
    int  enabled;
    int  flags;
    int  min_rate;
    int  max_rate;
    int  min_channels;
    int  max_channels;
    int  binding;
    int  rate_source;
    char handle[32];
    unsigned int nrates;
    unsigned int rates[20];
    oss_longname_t song_name;
    oss_label_t label;
    int  latency;
    oss_devnode_t devnode;
    int  next_play_engine;
    int  next_rec_engine;
    int  filler[184];
} oss_audioinfo;

The trick in Rust is to use #[repr(C)], libc::c_int and other libc::c_* types. If you put this right below use block, you have the whole code.

#[repr(C)]
struct AudioInfo {
    pub dev: libc::c_int,
    pub name: [libc::c_char; 64],
    pub busy: libc::c_int,
    pub pid: libc::c_int,
    pub caps: libc::c_int,
    pub iformats: libc::c_int,
    pub oformats: libc::c_int,
    pub magic: libc::c_int,
    pub cmd: [libc::c_char; 64],
    pub card_number: libc::c_int,
    pub port_number: libc::c_int,
    pub mixer_dev: libc::c_int,
    pub legacy_device: libc::c_int,
    pub enabled: libc::c_int,
    pub flags: libc::c_int,
    pub min_rate: libc::c_int,
    pub max_rate: libc::c_int,
    pub min_channels: libc::c_int,
    pub max_channels: libc::c_int,
    pub binding: libc::c_int,
    pub rate_source: libc::c_int,
    pub handle: [libc::c_char; 32],
    pub nrates: libc::c_uint,
    pub rates: [libc::c_uint; 20],
    pub song_name: [libc::c_char; 64],
    pub label: [libc::c_char; 16],
    pub latency: libc::c_int,
    pub devnode: [libc::c_char; 32],
    pub next_play_engine: libc::c_int,
    pub next_rec_engine: libc::c_int,
    pub filler: [libc::c_int; 184],
}

Although nobody can give you formula on how to write ioctl code in Rust based on C but there are some guidelines. Based on C code, you can tell which macro it uses so here’s a rough table.

C	Rust
_IO	ioctl_none
_IOR	ioctl_read
_IOW	ioctl_write_*
_IOWR	ioctl_readwrite

For a list of ioctl_write_* macros and full documentation refer to https://docs.rs/nix/latest/nix/sys/ioctl. I have to say it is easy to dive into something like this when you have one of the Rust advocates in FreeBSD as a work colleague, as he was really quick to spot my errors and generally guide me to a working code as a newbie Rustacean.

New Guitar

Mon, 09 Sep 2024 10:55:00 +0000

After 3 years in the making, I finally got my new guitar. I got Seymour Duncan Blackouts Metal for it. That pickup is insanely hot and until my new guitar was done, I used it in my Les Paul, so pickup is the same in both guitars. Here is a riff from our upcoming single with Neural Amp Modeler and Impulse Response instead of guitar amplifier, cabinet and microphone.

Gibson Les Paul Custom

Signature model

My impressions

I love Les Paul Custom for so long and I considered it the best guitar in the world. Together with EMG 81/85, it is insane. I can’t believe that I’m saying this, but there is something better than Les Paul, and it’s my new guitar. It is made by a long time friend of mine and solo guitarist in my band, so he knows what picking style I use, what bands I like, what amps I’m after. With somebody knowing me so well, he couldn’t but make a guitar better than Gibson. I love how it bites more than Les Paul. It will definitely be my sound on our upcoming single and album.

Just in case you’re wondering how it looks, it is black ESP Viper with “LUST” (the first word of the name of my band) instead of inlays.

Ratamahatta

Mon, 02 Sep 2024 10:17:00 +0000

I always mumbled Ratamahatta without understanding what’s it about. I did get literal translation from some friends, but I still didn’t understand what was it about. Nowadays I’m lucky that some of my colegues are Brazilians so I could ask for the deeper meaning of the words. The colegue in question is Fernando Seidler.

Biboca, garagem, favela
Biboca, garagem, favela!
Fubanga, maloca, bocada
Fubanga, maloca, bocada!

Maloca, bocada, fubanga
Maloca, bocada, fubanga!
Favela, garagem, biboca
Favela, garagem, biboca, porra!

So, these are words that make reference to slums or simple/dangerous places. Biboca is a word for a simple, humble place. Garagem is a garage. Favela is slum. Fubanga is a ugly, non-attractive woman. Maloca is a kind of indigenous habitation and Bocada is the place where people sell drugs.

Um, dois, três, quatro
Zé Do Caixão, Zumbi, Lampião
Zé Do Caixão, Zumbi, Lampião
Zé Do Caixão, Zumbi, Lampião
Zé Do Caixão, Zumbi, Lampião

Those 3 are some people from Brazil. Zé do Caixão is the character of a cult terror/bizarre genre. The actor is considered the father of Brazilian terror cinema. Zumbi makes reference to “Zumbi dos Palmares”, a guy who was a leader for slaves that ran away from their “masters” (I really hate the word “masters” in this context) and went to “Quilombo dos Palmares”, a settlement for escaped slaves. And Lampião was a bandit that is almost a myth: for the police he was a criminal, for the population at the time, a hero, so much that he received the nickname “Robin Hood da Caatinga”.

Vamos detonar essa porra
É, porra!

In a kind of literal translation:

“Let’s fuck this shit up
Yeah, fuck it”

For the end, here’s one formula: Ratamahatta = Ratos(rats) + Manhattan.

According to Carlinhos Brown, in an interview with UOL, the song was inspired by the rats of New York, that is, these animals do not only inhabit unhealthy places. “I had this title ‘Ratamahatta’ in my head and I needed to finish it. Max helped me. When I was in Manhattan, I was impressed by their rats. They look like the opossums here in Bahia. And opossums here go to the pot! But people said that the rats in Manhattan were not good to eat, because they were dirtier. That’s why the lyrics have the words “garagem” “biboca” and “favela”. The rat is not present only in the big cities. It is everywhere. It is at fubanga, maloca, bocada. Who is the rat? He is the recycling boss. This is what the song means: Talks about a Manhattan rat who goes around the world and connects with everybody.

OpenLDAP

Sun, 14 Apr 2024 17:43:00 +0000

I wanted to achieve 3 things:

Ability to enable/disable account and domain
Proper groups and membership
Authorization for services

I personally don’t like OLC or On-Line Config, as I like to do my things using Ansible. So here is the slapd.conf:

include		/usr/local/etc/openldap/schema/core.schema
include		/usr/local/etc/openldap/schema/cosine.schema
include		/usr/local/etc/openldap/schema/inetorgperson.schema
include		/usr/local/etc/openldap/schema/nis.schema
include		/usr/local/etc/openldap/schema/opendkim.schema
include		/usr/local/etc/openldap/schema/pmi.schema

pidfile		/var/run/openldap/slapd.pid
argsfile	/var/run/openldap/slapd.args

modulepath	/usr/local/libexec/openldap
moduleload	back_mdb
moduleload	memberof

overlay	    		memberof
memberof-group-oc	groupOfUniqueNames
memberof-member-ad	uniqueMember
memberof-refint		TRUE

TLSCACertificateFile /usr/local/etc/openldap/certs/chain.pem
TLSCertificateFile /usr/local/etc/openldap/certs/fullchain.pem
TLSCertificateKeyFile /usr/local/etc/openldap/certs/privkey.pem

security ssf=128 tls=1

access to attrs=userPassword
  by self write
  by anonymous auth

access to *
  by self write
  by users read
  by anonymous auth

database	mdb
suffix		"dc=ldap"
rootdn		"cn=root,dc=ldap"
directory	/var/db/openldap-data
index		objectClass,mail	eq
include		/usr/local/etc/openldap/slapd-secret.conf
include		/usr/local/etc/openldap/slapd-multimaster.conf

And this is the interesting part of the directory:

dn: dc=ldap
objectClass: domain
dc: ldap

dn: dc=account,dc=ldap
objectClass: domain
dc: account

dn: ou=meka.rs,dc=account,dc=ldap
objectClass: organizationalUnit
ou: meka.rs

dn: uid=meka,ou=meka.rs,dc=account,dc=ldap
objectClass: pilotPerson
objectClass: posixAccount
cn: Goran
sn: Mekić
uidNumber: 65534
gidNumber: 65534
homeDirectory: /var/mail/domains/meka.rs/meka
mail: meka@meka.rs
userClass: enabled
uid: meka

dn: dc=group,dc=ldap
objectClass: domain
dc: group

dn: cn=mail,dc=group,dc=ldap
objectClass: groupOfUniqueNames
cn: mail
uniqueMember: uid=meka,ou=meka.rs,dc=account,dc=ldap

dn: dc=service,dc=ldap
objectClass: domain
dc: service

dn: cn=postfix,dc=service,dc=ldap
objectClass: person
cn: postfix
sn: service
description: SMTP service

Let me ignore enable/disable of domain for a bit. Let’s just focus on accounts. In slapd.conf, every line with memberof string in it is for groups. By default, memberof module uses groupOfNames, but I think it is better to use groupOfUniqueNames, so it needs some extra configuration. Let’s see what it provides.

ldapsearch -x -Z -W -D cn=root,dc=ldap memberOf=cn=mail,dc=group,dc=ldap '*' 'memberOf'

. . .

dn: uid=meka,ou=meka.rs,dc=account,dc=ldap
objectClass: pilotPerson
objectClass: posixAccount
cn: Goran
sn: Mekić
uidNumber: 65534
gidNumber: 65534
homeDirectory: /var/mail/domains/meka.rs/meka
mail: meka@meka.rs
userClass: enabled
uid: meka
memberOf: cn=mail,dc=group,dc=ldap

. . .

So with memberOf filter, you can easily get members of a group. Notice that there’s '*' 'memberOf' at the end. That says “give me all attributes of an object, plus give me memberOf”. If you omit memberOf, it will not be displayed, although you requested all attributes. That is because it is dynamic attribute and it is returned only if explicitly requested. But there are few cases I found that gave me headache. For example, if you create the group, then configure slapd.conf to use it, it won’t work. I guess that something is triggered on creation and/or modification of a group that is not triggered in this scenario. Another anomaly I found is when I’m restoring from backup. For some reason, groups are created before accounts, so memberOf doesn’t work. Having dc=group,dc=ldap separated from dc=account,dc=ldap allows you to restore accounts before groups. Also, there is msuser.schema. I first thought that I need to include it to be able to use memberOf, but that is wrong. When I include this file, it defines memberOf and I guess that’s why it doesn’t work the way I wanted.

Look closely at ldapsearch output and notice userClass: enabled. If extend the previously used filter, you get something like this:

ldapsearch -x -Z -W -D cn=root,dc=ldap (&(userClass=enabled)(memberOf=cn=mail,dc=group,dc=ldap)) '*' 'memberOf'

Output will pretty much be the same, only difference is which accounts will be listed.

Enabling or disabling domain is partially working. For example, postfix has a filter for domains, but dovecot does not, while ejabberd can’t use LDAP for list of domains. Depending on the capabilities of a service, you might or might not achieve this. For example, solution I use for ejabberd is to have group, just like for the accounts, but for domains called enabled and use same memberOf filter to get the list. Then I use that info in Ansible to provision configuration file.

For authorization of services I chose not to use dc=account,dc=ldap for base, but cn=<service>,dc=service,dc=ldap. That way services can not interfere with user accounts, but beside different base, those are just like normal accounts.

For more context on how I use OpenLDAP, take a look at my set of services for communication as it might give you a broader picture what I’m trying to solve.

DIY Sound

Sat, 12 Aug 2023 10:46:25 +0000

Sometimes you just hate what industry has don with the sound and start building your own stuff. For one, I didn’t like open source DAW alternatives as all of them are missing something: Arour’s MIDI is terrible for electronic music, loop recording is missing in Muse, ZRythm is stereo only and I couldn’t find how to work with QTractor and DrumGizmo. To learn about audio and DSP, I started work on Maolan. Today Maolan is still extremely experimental, but I learned a lot and I plan to work on it as free time permits. I could have started with JACK as first supported backend, but I decided to go with FreeBSD’s OSS and really understand what’s going on there. Today I’m on the verge of understanding it all perfectly from user space perspective. I am not as good kernel developer, but I do intend to optimize FreeBSD’s OSS and other code that is part of audio stack. For example, currently USB audio driver has minimal buffer size of 2 miliseconds. Not bad, but USB docs say it works in 1ms frames with 8 subframes which are 125us or microseconds. Lowering that will be piece of cake, but I’m waiting for 14-STABLE to branch off so I don’t sneak in buffer limit which is not tested enough. As OSS, at least in FreeBSD, uses double buffer: one user facing, one hadware facing, it would be interesting to see what optimizations are possible if we would expose controls for hardware facing buffer. You see where I’m going with this. There’s work to be done to make FreeBSD more real-time.

But software and PC in general is just a half the story. Audio interface and hardware audio devices are the other half. Today USB audio is hard to work with on any operating system. That 1ms frame of USB makes sure that latency is at least 1ms. Using PCIe lowers that, but developing such card is way too hard for me. My plan/workaround is to build USB sound card with very high sampling rate. I will start with 384kHz and some of-the-shelf solutions to support it. List of items I’m waiting to arrive:

If you look at AD/DA converters, you’ll notice that they can do 32bit @ 768kHz. That will be my next step, after all these parts start working together. The ultimate goal is to make DIY high end audio interface that has one less digit in the price. For example, if I didn’t need 2 programming adapters, that would be more than 200e less on the price. The reason I’m getting it anyway is because I can’t solder that precisely, yet. I would love to program some MCU and USB PHY using NuttX and using more general board instead of MCHStreamer, but I’m not there yet. In order to get to 768kHz, I have to replace it anyway, so I’m sure I will get there eventually. If all that work gives latency less than 6ms, I finally have real-time using USB. Also, if I have all that, how hard will it be to make PCIe alternative? I know it will be much faster, and with it I might go to latencies like 2ms which give you freedom to patch things any way you want, like microphone into the interface, have it recorded and pushed to output which is connected to hardware vocal processor and then again back to the interface, and then … The point is that every time you go through sound card you add those 2ms which means you can do it 3 times and still be called real-time. And maybe I won’t do that but build digital mixer which will route sound internally and it won’t even need a computer do work. Of course, I will have USB interface on it, because I do want to record it in the end.

Reading about AD/DA, I stumbled upon DSD format which they say is superior to PCM. Let me briefly explain. PCM is what we are usually thought is digital sampling. It’s how CD works, for example. Every 1/44100 of a second, level of signal is measured and it is represented with a number. That’s what the name stands for: Pulse Code Modulation or PCM for short. DSD works differently. I only uses one bit to represent the sample. Weird, right? The idea is that one bit is enough to represent if the current sample is higher or lower than previous one. Done at a high frequency, for example >10MHz, you can easily see that it follows audio signal more accurately. At least that what I read on the Internet. Now here’s an idea. If I get my sound interface working in DSD mode, and write a patch for FreeBSD OSS so it handles DSD natively, and use DSD WAV files to record the input, I’m hoping to capture audio more accurately. I don’t know how much difference it will make but today that’s the best what the industry can offer, yet can’t use it as no interface today is built like that. I mean, I’m not the first to come up with this idea, for start Denoy used same AD/DA until the shortage, so the components are field tested, I’m not interested in revolutionizing anything. What I mean is that the components I listed are on the market for years and are being used by audio companies for years, so I know I can do it, it’s just my first prototype so I can’t really say how or when.

My dear reader, I have a question. Would you support such a project by funding it? If no, why? If yes, what amount? To be more precise, would you fund my work on everything I just described if I promise that USB audio interface I just described will cost around 200e (with current prices) if you exclude shipping, you would have to solder it and get parts but you get the whole KiCAD project, all code open source and based on NuttX? If you just had “yes to all” moment, let me ask additional question. Once all this is done, would you fund work to make it USB 3 compatible? As USB 3 has much higher bandwidth, the interface could have much more channels, in or out. I’m talking about 32in, 32out, if your PC can cope with it.

Please let me know what you think. I prefer email as it’s easiest to sort and archive of all communication channels, but you’re free to pick whatever you like to contact me.

FreeBSD PF and Dummynet

Thu, 17 Nov 2022 11:57:00 +0000

In 14-CURRENT there is now support for dummynet in PF. That means that you can slow down packets based on some criteria. MacOS users probably know how this works as that OS got support for PF+dummynet years ago. For example, you can do the following in /etc/pf.conf:

pass in quick inet from 192.168.1.1 to any dnpipe 1

That way all traffic from 192.168.1.1 will go through dummynet pipe. To create and configure the pipe you use:

dnctl pipe 1 config bw 300KByte/s

I gave it a really low bandwidth because I want it to be really noticeable if packets are going through dummynet or not. You can change the pipe’s bandwidth by using the same command just changing the numbers. Note that K and B have to be upper case. You can, of course, use all the usual suffixes like M and G but note that dummynet has a limit on the bandwidth it can configure.

To see what is configured you can use:

dnctl pipe show

With dnctl, show is an alias for list, so you will get the same results using either.

I didn’t make it so that packets generated on the machine itself be processed by dummynet, but it might be intentional. What does work is VNET so your jails can have different PF and dummynet configuration than the host.

One annoying thing is that there is no dnctl rc.d service, so you will probably write dnctl commands in something like rc.local or something. I do plan to create rc.d service for myself and when I learn more about dummynet, to publish it hoping it will become part of base so we can easily configure it.

Update

The dnctl service is now available. If enabled it reads /etc/dnctl.conf which has IPFW-like syntax and when only dummynet module is loaded, it will issue an error on non-dummynet IPFW statements.

FreeBSD Dual Stack Firewall

Tue, 16 Aug 2022 11:11:35 +0000

The idea is very simple: everything works. But what is “everything” in this case? I want my host and jails to have IPv4 and IPv6 address, register those addresses in DNS and all to be accessible in the network and via Internet. To make my life easier, I programmed Reggae to initialize rtadvd, jail with isc-dhcpd for IPv4 and IPv6 and DNS, pf with the base configuration.

Host

If there were no jails, the configuration would be simple: just use dhcpcd for everything. There’s not even need for some special firewall rules. To do that, you need to run:

pkg install dhcpcd
echo 'dhclient_program=/usr/local/sbin/dhcpcd' >/etc/rc.conf.d/network

On next reboot or netif start all your DHCP ifaces will use dhcpcd and it will configure DHCPv4, SLAAC and DHCPv6. Admitedly you will have two IPv6 addresses: one SLAAC configured and one acquired through DHCPv6. The DHCP addresses will be registered in DNS. That’s basically it.

Jail

In the jail itself it is the same as host: install dhcpcd and use it and DHCP addresses are registered in DNS. If all you need is some kind of dual stack, that’s it, but if you need it on a server with jails, stuff gets a little bit more complex. First, to achieve that host and jails get the address from the same router, you need to bridge your physical interface and epairs.

cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0"
ifconfig_bridge0_ipv6="inet6 fd10:6c79:8ae5:8b91::5 -ifdisabled auto_linklocal"

The example uses “private IP range” in IPv4 terms, or unique local addresses as IPv6 terminology defines them. There are two main problems with this setup: DHCPv6 will not work and firewall will do too much. For DHCPv6 the reason it doesn’t work is that you have to allow dhcpv6-client messages to arrive. You didn’t need it for host-only setup, but to allow those messages to reach jails, you need.

pass in quick inet6 proto udp from fe80::/10 port dhcpv6-server to fe80::/10 port dhcpv6-client

When you put egress interface (igb0 in my case) into bridge, it will see all traffic for jails, too, so you have to allow packets which are not destined to the host. To achieve that, pf offers .

block in log from any to <self>

That way you are filtering everything for the host, but leave jail traffic alone.

Router

Maybe the easiest thing to do is converting Reggae setup into your router. All you need to do is add physical interfaces like re1 and wlan0, to your bridge and that’s it. As all services inside network jail (the one with DHCP and DNS) are listening on epair which is part of bridge, no other actions is needed.

Reggae

Reggae will initialize your network, services and network jail in dual stack mode. You can disable IP version by setting USE_IPV4=no or USE_IPV6=no, but it will issue an error if you disable both. It will also write /etc/pf.conf if one doesn’t already exist and setup local_unbound, so you should be all set after initializing Reggae the usual way.

FreeBSD and YubiKey

Wed, 20 Jul 2022 10:27:00 +0000

Install and initialize the services as root:

pkg install ccid opensc pcsc-lite
sysrc pcscd
service pcscd restart

Start ssh agent and add provider to it as user:

eval $(ssh-agent)
ssh-add -s /usr/local/lib/opensc-pkcs11.so

That’s it, SSH should work with YubiKey now.

FreeBSD Linuxulator

Fri, 01 Jul 2022 00:21:00 +0000

As some applications are Linux-only, it is very handy to have linuxulator available. In short, I just followed linux browser install and added slack and viber with pulseaudio setup. Although pulseaudio setup contains linux browser install, it is the most important part. Once the browser was installed I copied the scripts to make slack and viber working. After download of .deb file, chroot to linux directory and install it.

cp <slack>.deb /compat/ubuntu
chroot /compat/ubuntu /bin/bash
dpkg -i <slack>.deb
apt-get install -f

On host, script is needed in PATH so it can be executed just like a normal application, so /usr/bin/slack looks like this:

#!/bin/sh

get_pa_sock_path()
{
  PA_SOCK_PATH=$(sockstat | awk -v me=$(whoami) -F'[ \t]+' '
    $1 == me && $2 == "pulseaudio" && $6 ~ /native/ {
      print $6;
      exit 0
    }'
  )
}

get_pa_sock_path
[ -S "$PA_SOCK_PATH" ] && export PULSE_SERVER=unix:$PA_SOCK_PATH

/compat/ubuntu/bin/slack $@

Next, we need to create /bin/slack inside ubuntu chroot:

#!/compat/ubuntu/bin/bash
#
# chrome wrapper script from patovm04:
# https://forums.freebsd.org/threads/linuxulator-how-to-run-google-chrome-linux-binary-on-freebsd.77559/
#
export SLACK_PATH="/usr/bin/slack"
export SLACK_WRAPPER="$(readlink -f "$0")"
export LD_LIBRARY_PATH=/usr/local/steam-utils/lib64/fakeudev
export LD_PRELOAD=/usr/local/steam-utils/lib64/webfix/webfix.so
export LIBGL_DRI3_DISABLE=1
exec -a "$0" "$SLACK_PATH" --no-sandbox --no-zygote --test-type --v=0 "$@"

Final change is to /usr/share/applications/slack.desktop inside ubuntu chroot:

[Desktop Entry]
Name=Slack
StartupWMClass=Slack
Comment=Slack Desktop
GenericName=Slack Client for Linux
Exec=/bin/slack %U
Icon=/usr/share/pixmaps/slack.png
Type=Application
StartupNotify=true
Categories=GNOME;GTK;Network;InstantMessaging;
MimeType=x-scheme-handler/slack;

Note: Exec line is wrong by default for linuxulator environment. The executable needs to be our script /bin/slack, not the slack binary itself /usr/bin/slack.

The trick is to execute pulseaudio --daemonize on FreeBSD host before starting slack. I got audio and webcam working. Of course, viber setup is the same.

FreeBSD Dual Stack Jails

Sat, 22 Jan 2022 09:35:00 +0000

There are 3 ways to have IPv6 in VNET jails that I’m going to describe. In terms of IPv4, those would be:

private address
public address
public address behind router

Private addresses in IPv6 world are called “unique local”, and they start with “fd” in hexadecimal notation. Please note that there are also site-local addresses, but they are deprecated. Something like a MAC address is called “link local address” and they start with “fe80” in hex. In this example, all addresses starting with 2001 are global or in IPv4 notation, they are public. These are not the only types of addresses, nor addresses starting with 2001 are the only global ones. I’m just trying to keep it simple by narrowing it down to 3 types.

Jail Setup

As CBSD renames jail’s end of epair to eth0 writing rules/config is simplified. In all jails the setup of /etc/rc.conf is:

ifconfig_eth0_ipv6="inet6 -ifdisabled accept_rtadv auto_linklocal"
rtsold_enable="YES"

CBSD will run DHCP client before running init, so that part is taken care of if you choose REALDHCP as address assigning method.

Unique Local Addresses

I use this setup for development as I don’t want everything I do to be public all the time. The principles are the same as with IPv4: assign jails private addresses and then NAT them to the world. The relevant portion of /etc/rc.conf is:

ifconfig_re0="DHCP"
ifconfig_re0_ipv6="inet6 -ifdisabled auto_linklocal accept_rtadv"
ipv6_gateway_enable="YES"
ipv6_defaultrouter="fe80::5a9c:fcff:fe10:6c2c%re0"
rtsold_enable="YES"

# CBSD
ifconfig_bridge0_name="cbsd0"
ifconfig_cbsd0="inet 172.16.0.254 netmask 255.255.255.0 description lagg0"
ifconfig_cbsd0_alias0="inet 172.16.1.254 netmask 255.255.255.0"
ifconfig_cbsd0_ipv6="inet6 fd10:6c79:8ae5:8b91::1 -ifdisabled auto_linklocal"

As I use CBSD and Reggae, I like to create dedicated bridge interface for it and rename it so it’s easier to tell what is what. I also have two IPv4 ranges: one for DHCP assigned addresses and one for CBSD generated ones. For IPv6 I have one address from the same range jails get their addresses. The configuration of /etc/rtadvd.conf:

cbsd0:addr="fd10:6c79:8ae5:8b91::"

For NAT /etc/pf.conf is:

# Macros and tables
ext_if = "lagg0"
table <cbsd> persist

# Options
set block-policy drop
set skip on lo0

# Normalization
scrub in all

# NAT
rdr-anchor "cbsd/*" on $ext_if
nat on $ext_if inet from <cbsd> to any -> ($ext_if)
nat on $ext_if inet6 from cbsd0:network to any -> ($ext_if:0)

# Quick rules
antispoof quick log for ($ext_if)

# Rules
block in log on $ext_if
pass out
pass proto tcp to any port ssh
pass proto { icmp, igmp, icmp6 }

There is a bit more than a bare minimum, but the important lines are those starting with “nat”. First rule is for IPv4 and it will NAT for all addresses CBSD/Reggae puts in table. Second rule is for IPv6 and it is important to use $ext_if:0 not just $ext_if because it will otherwise try to NAT using all IPv6 addresses. In this case it would use the proper address as well as link-local one.

Global Unicast Address

This setup should be almost the same as the previous one. Keep in mind that in the following setup re0 and cbsd0 must use same prefix, or in IPv4 terms: they have to be in the same network. In practice, that means that both interfaces must have IPv6 address which starts with 2001:aaaa:bbbb:cccc:. Equally important is to not add re0 to cbsd0 bridge as that would make local DHCP server running in a jail leak out through re0 towards the rest of the physical network.

/etc/rc.conf:

ifconfig_re0="DHCP"
ifconfig_re0_ipv6="inet6 -ifdisabled auto_linklocal accept_rtadv"
ipv6_gateway_enable="YES"
ipv6_defaultrouter="fe80::5a9c:fcff:fe10:6c2c%re0"
rtsold_enable="YES"

# CBSD
ifconfig_bridge0_name="cbsd0"
ifconfig_cbsd0="inet 172.16.0.254 netmask 255.255.255.0 description lagg0"
ifconfig_cbsd0_alias0="inet 172.16.1.254 netmask 255.255.255.0"
ifconfig_cbsd0_ipv6="inet6 2001:aaaa:bbbb:cccc::1 -ifdisabled auto_linklocal"

/etc/rtadvd.conf:

cbsd0:addr="2001:aaaa:bbbb:cccc::"

/etc/pf.conf:

# Macros and tables
ext_if = "lagg0"
table <cbsd> persist

# Options
set block-policy drop
set skip on lo0

# Normalization
scrub in all

# NAT
rdr-anchor "cbsd/*" on $ext_if
nat on $ext_if inet from <cbsd> to any -> ($ext_if)

# Quick rules
antispoof quick log for ($ext_if)

# Rules
block in log on $ext_if
pass out
pass proto tcp to any port ssh
pass proto { icmp, igmp, icmp6 }

Global Unicast Address Behind Router

This setup is mostly for the server behind a router. The idea is to put all physical (in this case one) and virtual (in this case epair interfaces) into the same bridge. Bridge acts like a switch, so the network will behave like we somehow plugged all physical and virtual interfaces into the same switch. In practice it means that DHCP and rtadv/rtsol packets will got to/from the router, directly. As it is a server, all configuration is static, so there’s no rtsold/rtadvd present.

/etc/rc.conf:

ifconfig_igb0="inet 192.168.111.201 netmask 255.255.255.0"
ifconfig_igb0_ipv6="inet6 2001:aaaa:bbbb:cccc::4 -ifdisabled auto_linklocal"
ipv6_defaultrouter="fe80::5a9c:fcff:fe10:6c2c%igb0"
defaultrouter="192.168.111.254"

# CBSD
cloned_interfaces="bridge0"
ifconfig_bridge0_name="cbsd0"
ifconfig_cbsd0="description igb0 addm igb0"

/etc/pf.conf:

# Macros and tables
ext_if = "igb0"
dhcp_ports = "{ bootps, bootpc }"
table <cbsd> persist

# Options
set block-policy drop
set skip on lo0

# Normalization
scrub in all

# Rules
block in log on $ext_if
pass out
pass in from <cbsd> to any
pass in on $ext_if from any to $ext_if:network
pass proto tcp to any port ssh
pass proto { icmp, igmp, icmp6 }
pass in proto udp from any to any port $dhcp_ports

Note that in this setup NAT and antispoof are missing while it’s essential to have pass in on $ext_if from any to $ext_if:network. NAT is not needed as router will do it in this setup and antispoof is not applicable here because it practically says “any packet with source address from igb0 network coming from interface other than igb0 should be blocked”. That’s the problem because VNET jails will have epair interfaces using the same address range and antispoof would block those packets as well because they physically pass through igb0 and are visible. The extra pass rule is for the same reason.

FreeBSD Dual Stack Network

Sat, 15 Jan 2022 22:55:00 +0000

Idea is to build dual stack network, which means working IPv4 and IPv6. Support on router and client side is needed for network to work without glitches. For example, switching from ethernet to WiFi should be seamless. On top, as my ISP is not providing IPv6, I’ll show you how tunnels like Hurricane Electric work, which in layman terms means “how to have public IPv6 addresses on all my devices although ISP doesn’t provide it”. If you have native IPv6 support from your provider, that’s great and just use it, otherwise you can use Huricane Electric Tunnel Broker to setup IPv6.

Router

The following is the simplified picture of a network. There is a switch missing and WiFi is just one line between laptop and router but it’s good enough for explaining what will be configured. On the left of the router is IPv4 connection that ISP provides, and on the left is IPv6 connection that HE tunnel provides. There are few services to keep everyone happy, like DHCP server inside CBSD based jail, Router Advertisment daemon (rtadvd) to “disperse” IPv6 addresses, and so on.

.------.   .--------.   .---------.
| IPv4 |<->| Router |<->| HE IPv6 |
.______.   .________.   ._________.
               /|\
              / | \
             /  |  \
            /   |   \
           /    |    \
          /     |     \
 .--------. .---------. .---------.
 | Laptop | | Seerver | | Desktop |
 |________| |_________| |_________|

There are are 3 ethernet ports on APU and one WiFi interface. One ethernet port is used for connection to ISP, two remaining ports and WiFi are bridged into one interface. There are few services on the router to make it work:

Router Advertisment (rtadvd)
Host AP daemon (hostapd)
DHCP in jail (isc-dhcpd)
Unbound for DNS (local_unbound)
Firewall (pf)
PPP daemon based on netgraph (mpd5)
Jail management (cbsd)
Protection from ssh brute force (blacklistd)

/etc/rc.conf:

# Network
cloned_interfaces="bridge0 bridge1"
ifconfig_re0="inet 192.168.100.254 netmask 255.255.255.0"
ifconfig_re1="up"
ifconfig_re2="up"
ifconfig_bridge0_name="cbsd0"
ifconfig_bridge1_name="lan"
ifconfig_lan="addm re1 addm re2 addm wlan0 stp re1 stp re2 stp wlan0"
ifconfig_lan_alias0="inet 192.168.111.254 netmask 255.255.255.0"
ifconfig_lan_ipv6="inet6 2001:aaaa:bbbb:cccc::3 auto_linklocal -ifdisabled"
wlans_ath0="wlan0"
ifconfig_wlan0_ipv6="inet6 -ifdisabled"
create_args_wlan0="wlanmode hostap"
ifconfig_wlan0="txpower 50 channel 149 up"
hostapd_enable="YES"
rtadvd_enable="YES"
gateway_enable="YES"
ipv6_gateway_enable="YES"
local_unbound_enable="YES"
local_unbound_tls="NO"

# HE IPv6 tunnel
gif_interfaces="gif0"
gifconfig_gif0="MyIPv4 HE-IPv4"
ifconfig_gif0_ipv6="inet6 2001:aaaa:bbbb:cccc::2 2001:aaaa:bbbb:cccc::1 prefixlen 128"
ipv6_defaultrouter="2001:aaaa:bbbb:cccc::1"

# Firewall
pflog_enable="YES"
pf_enable="YES"

# CBSD
ifconfig_cbsd0="inet 172.16.0.254 netmask 255.255.255.0 description ng0"
ifconfig_cbsd0_alias0="inet 172.16.1.254 netmask 255.255.255.0"
cbsd_workdir="/usr/cbsd"
cbsdrsyncd_enable="YES"
cbsdrsyncd_flags="--config=/usr/cbsd/etc/rsyncd.conf"
cbsdd_enable="YES"
rcshutdown_timeout="900"

The CBSD portion is present on all machines as I use it to manage my jails, but I won’t repeat it in every configuration.

To turn my WiFi into AP I use hostapd with the following configuration.

/etc/hostapd.conf

interface=wlan0
debug=1
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel
ssid=myssid
wpa=2
wpa_passphrase=Secrit
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP

As my ISP uses PPPoE I have to use mpd5. It performs much faster than PPP included in base. I could get about 200Mbit/s with the one from base, while I have full 400Mbit/s with mpd5.

/usr/local/etc/mpd5/mpd.conf

default:
  load mts

mts:
  create bundle static B1
  set iface route default
  set iface enable tcpmssfix
  set ipcp ranges 0.0.0.0/0 0.0.0.0/0
  
  create link static L1 pppoe
  set link action bundle B1
  set auth authname isp_username@open.telekom.rs
  set auth password isp_password
  set link max-redial 0
  set link keep-alive 10 60
  set pppoe iface re0
  set pppoe service ""
  open

/etc/pf.conf

# Macros and tables
ext_if = "ng0"
modem_if = "re0"
modem = 192.168.100.1

# Options
set block-policy drop
set skip on lo0

# Normalization
scrub in all

# NAT
nat on $ext_if inet from lan:network to any -> ($ext_if)
nat on $modem_if inet from lan:network to any -> ($modem_if)

# Quick rules
antispoof quick for ($ext_if)
anchor "blacklistd/*" in on $ext_if

# Rules
block in log on $ext_if
pass out
pass proto tcp to any port ssh
pass proto { icmp, igmp, icmp6 }

The default prefix is 64, so the configuration is short. Note that you can use two forms to assign value to an attribute: attribute#value which is the same as attribute=“value”.

/etc/rtadvd.conf

lan:addr="2001:aaaa:bbbb:cccc::"

In jail DHCP for IPv4 is running with the following configuration in /usr/local/etc/dhcpd.conf:

server-identifier my.domain.tld;
authoritative;
log-facility local7;
omapi-port 7911;


subnet 172.16.0.0 netmask 255.255.255.0 {
  option domain-name "my.domain.tld";
  option domain-name-servers 172.16.0.254;
  range 172.16.0.1 172.16.0.200;
  option routers 172.16.0.254;
  option broadcast-address 172.16.0.255;
  default-lease-time 3600;
  max-lease-time 7200;
  on commit {
    set clientIP = binary-to-ascii(10, 8, ".", leased-address);
    set clientHost = pick-first-value(option fqdn.hostname, option host-name, "");
    execute("/usr/local/bin/dhcpd-hook.sh", "add", clientIP, clientHost, "my.domain.tld");
  }
  on release {
    set clientIP = binary-to-ascii(10, 8, ".", leased-address);
    set clientHost = pick-first-value(option fqdn.hostname, option host-name, "");
    execute("/usr/local/bin/dhcpd-hook.sh", "delete", clientIP, clientHost, "my.domain.tld");
  }
  on expiry {
    set clientIP = binary-to-ascii(10, 8, ".", leased-address);
    set clientHost = pick-first-value(option fqdn.hostname, option host-name, "");
    execute("/usr/local/bin/dhcpd-hook.sh", "delete", clientIP, clientHost, "my.domain.tld");
  }
}

The configuration could be much smaller without hooks, but this way you have enough information how I register jails in DNS. This part will be detailed on cbsd.io.

Laptop

It’s important to set wlan0 MAC address to be the same as your ethernet. In my case it is em0. For some reason, rtsold which comes with the FreeBSD base doesn’t work stable and my current workaround is to add ipv6_defaultrouter. It does kinda defeat the purpose of software called “router advertisement”, but until this bug is resolved, it is good enough for me, as I have small number of machines. You might think I could use dhcpcd for IPv4 and IPv6, but there’s a problem with that: no matter how I configure jails, they just can’t get IPv6 when I’m using dhcpcd on the host. More on that in a later post.

/etc/rc.conf:

# Network
cloned_interfaces="lagg0"
wlans_iwn0="wlan0"
ifconfig_wlan0="ether f0:de:bb:aa:c2:2a WPA up"
ifconfig_em0="up"
create_args_wlan0="country US regdomain FCC"
ifconfig_lagg0="laggproto failover laggport em0 laggport wlan0 DHCP"
ifconfig_lagg0_ipv6="inet6 -ifdisabled accept_rtadv auto_linklocal"
gateway_enable="YES"
ipv6_gateway_enable="YES"
ipv6_defaultrouter="2001:aaaa:bbbb:cccc::3"

Server

On server I like to set static IPv4 and IPv6 addresses.

/etc/rc.conf:

ifconfig_igb0="inet 192.168.111.201 netmask 255.255.255.0"
ifconfig_igb0_ipv6="inet6 2001:aaaa:bbbb:cccc::4 -ifdisabled auto_linklocal"
ipv6_defaultrouter="fe80::5a9c:fcff:fe10:6c2c%igb0"
defaultrouter="192.168.111.254"

Not that “%igb0” means something like “search for this link-local address on igb0 interface”.

FreeBSD and IPv6 Tunnel

Sun, 19 Dec 2021 21:48:00 +0000

Huricane Electric is one of IPv6 tunnel providers and this post is specifically about setting up FreeBSD router with a tunnel and the configuration for IPv6 autoconfig caller Router Advertisment (RS, on the router) and Router Solisticion (RS, on desktop or laptop). If you know how DHCP for IPv4 works, you will find it a bit weird how IPv6 works. First, in IPv6 world what is called DHCPv6 is closer to what DHCPv4 does than RA/RS. Here, RA/RS setup will be described. The end result is that router and clients have access to IPv4 and IPv6 at the same time, or in how it’s called “Dual Stack”.

Router

Although Huricane Electric will give you exact commands/config for operating system of your choice, I’m showing here whole configuration for completeness. The address range fd12:c09a:85be:4851:: is just a dummy one, you should use the one HE provides.

In /etc/rc.conf:

ifconfig_re1="up"
ifconfig_re2="up"
ifconfig_bridge1_name="lan"
ifconfig_lan="addm re1 addm re2 addm wlan0 stp re1 stp re2 stp wlan0"
ifconfig_lan_alias0="inet 192.168.0.1 netmask 255.255.255.0"
ifconfig_lan_ipv6="inet6 fd12:c09a:85be:4851::3 auto_linklocal -ifdisabled"

# HE IPv6 tunnel
gif_interfaces="gif0"
gifconfig_gif0="<MyIPv4> <HEIPv4>"
ifconfig_gif0_ipv6="inet6 fd12:c09a:85be:4851::2 fd12:c09a:85be:4851::1 prefixlen 128"
ipv6_defaultrouter="fd12:c09a:85be:4851::1"

In /etc/rtadvd.conf:

lan:\
  :addrs#1\
  :addr="fd12:c09a:85be:4851::"\
  :prefixlen#64\
  :tc=ether\
  :rltime#0\
  :rdnss="fd12:c09a:85be:4851::3"\
  :dnssl="meka.rs"

In /etc/rc.conf.d/rtadvd:

rtadvd_enable="YES"
rtadvd_interfaces="lan"

Desktop/Laptop

In /etc/rc.conf:

cloned_interfaces="lagg0"
wlans_iwn0="wlan0"
ifconfig_wlan0="ether f0:de:f1:64:2c:3b WPA up"
ifconfig_em0="up"
create_args_wlan0="country US regdomain FCC"
ifconfig_lagg0="laggproto failover laggport em0 laggport wlan0 DHCP"
ifconfig_lagg0_ipv6="inet6 accept_rtadv -ifdisabled"
ipv6_defaultrouter="fd12:c09a:85be:4851::3"
rtsold_enable="YES"

Configuration is based on Router and Laptop on FreeBSD. It can be simplified if you use only wlan, which probably means you don’t need bridge at all hence you should configure wlan0, for example.

FreeBSD Audio

Tue, 12 Oct 2021 10:00:00 +0000

I started using FreeBSD in 2016 as a dual-boot with Linux. The reason was that at the time Linux provided no support for real-time threads and preemptive scheduling but did provide a wider choice of audio applications. Today, as the FreeBSD audio ecosystem improved a lot, I am a happy single-OS user.

You may ask yourself “why is being fast so important to audio” and the answer is “it isn’t most of the time”. To be more precise, from a listener’s perspective, it is the same if sound is late 5 seconds or 1 second, as long as the period of “being late” is the same for every sample up to a few microseconds. If samples have variable latencies, the original sound will be distorted by samples swinging around their precise time. So for playback, it’s not important to be fast, it’s important to be right on time for every sample to avoid distortions caused by samples being too early or too late, which is commonly called “jitter”.

On the other hand, a musician playing through a FreeBSD machine is interested in both: speed of processing and precision. We need to have in mind that all samples first have to be processed by hardware, then comes the driver and in the end user space programs. When every device’s output is late compared to input, that is called latency. If latency is 5ms and lower, it is impossible for most people to tell there’s any latency at all. Latency is important to musicians because of the feedback: hearing a sound that has 50ms latency can confuse the musician to a point where they can not play any more.

FreeBSD at the time had really nice real-time support but not so nice audio ecosystem and almost non-existing MIDI one. You must be wondering why I switched to something half-baked when Linux at the same time had awesome audio/MIDI support. It’s those jitters.

Although I had every program I could wish for on Linux, FreeBSD simply didn’t have jitters, and it matters a lot! Sometimes jitters would cause sudden and loud pops and that made studio recording more challenging. So having FreeBSD working perfectly, but unfortunately with not so much app support, was better for me than having all the apps I want and no way of being sure the sound will be recorded properly.

Today we have a different situation thanks to Hans Peter Selasky who wrote cuse, virtual_oss, USB stack and snd_uaudio and Yuri who ported around 1500 applications to FreeBSD, among which is a huge number of audio/MIDI apps. To be honest, virtual_oss existed in 2016, but it wasn’t as versatile and stable as today. The reason why anyone would want virtual_oss is to make audio routing easy by having a virtual sound card which knows how to route the signal while user space applications are unaware of it and they just use FreeBSD sound(4) API. There are numerous other features of virtual_oss that can come handy like mixing, compressing and EQ in user space, but audio routing and splitting/merging one card to many virtual ones or combining input and output from different devices is the most common use case, like having bluetooth headphones and USB microphone, so virtual_oss is required more and more outside of recording studio and high-end sound setups.

One sound system must contain an API to work with audio, MIDI and mixer. Open Sound System (or OSS for short) on FreeBSD is no exception to this rule, so there are 3 most common devices for every API: /dev/dsp, /dev/midi and /dev/mixer. The most basic usage for /dev/dsp is to open(2) it, use ioctl(2) to configure sample rate and format and then use read(2) and write(2) for recording and playback. It is somewhat similar for /dev/midi, but /dev/mixer is all about control and no samples, so it mostly uses ioctl(2) to operate. As every system needs metadata, /dev/sndstat is used for OSS. If you cat /dev/sndstat, you can find some information about your DSP and PCM devices. You can use sysctl hw.snd.verbose=2 to get even more information out of /dev/sndstat. Parsing this file as text is the only way to get the list of your sound devices that works across multiple FreeBSD versions. Ka Ho Ng used this technique in patch for cubeb, which is a sound library used by Mozilla, hence adding OSS and virtual_oss support to Firefox and other products by Mozilla. Today Ka Ho Ng is FreeBSD developer who implemented nvlist(9) based API to enumerate devices, or in simple terms: list hardware and virtual sound devices using nice API.

User space applications and libraries are growing and apps that I would like to mention that have been ported to FreeBSD are Ardour, Muse Sequencer, Zrythm, Drumgizmo, EQ10Q, Calf and Invada plugins. I use most of those in my studio on a regular basis and I have to admit I’m impressed how stable they work given that most developers did not develop with FreeBSD in mind.

Today in a studio, snd_uaudio and ports/packages will cover 99% of everyone’s needs. Few years back I talked to Benedict Reuschiling and he said “we never advocated FreeBSD for audio before”, so I’m sure everyone in the community is happy with advancements made in just a few years. I keep mentioning studio setups as they are more complex and demanding, but what about laptops and desktops whose sole purpose is not audio? That’s the beauty of FreeBSD audio and virtual_oss: if it works for complex setup, it works even better for simple one. All sample rate and format is taken care of per application while virtual_oss itself knows how to use real-time threads. Ideally, all that is needed is OSS support in applications.

For FreeBSD audio and DSP developers the situation is becoming increasingly better through newer APIs and more convenient development environments. For example, it is already possible to do all your development in jail if that jail has access to proper /dev/dspN. To achieve that following /etc/devfs.rules can be used:

[audio=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'dsp*' unhide mode 0666
add path 'vdsp*' unhide mode 0666
add path '*midi*' unhide mode 0666
add path 'mixer*' unhide mode 0666
add path 'sndstat' unhide mode 0666

That is not new for FreeBSD, but it does bring one interesting use case: running tests inside jail. To do just that maybe it means you would hear weird sounds on your speakers when all you wanted is an end-to-end test of your audio application. Virtual_oss has different backends where hardware DSP is just one of them. Another example of backend is dummy, which allows virtual_oss to run without connection to any real hardware. To do that you can configure /etc/rc.conf in the following way:

virtual_oss_enable="YES"
virtual_oss_configs="dsp dummy"
virtual_oss_dsp="-T /dev/sndstat -S -i 8 -C 18 -c 18 -r 48000 -b 32 -s 768 -f /dev/dsp0 -c 2 -w dsp.wav -d dsp -t dsp.ctl"
virtual_oss_dummy="-T /dev/sndstat -S -i 8 -C 2 -c 2 -r 48000 -b 32 -s 768 -f /dev/null -c 2 -w vdsp.wav -d vdsp -t vdsp.ctl"

There are two configs that are fairly similar. For dsp, one device with 18 in/out channels is configured as /dev/dsp which have 2 channels. Also, the 2 channel device is set as /dev/dsp which is the default OSS device, so applications which do not handle non-stereo cards will have no problems.

For those apps that know how to list sound devices and use channels properly /dev/vdsp is at their disposal. One unusual device is /dev/dsp.wav which you can use for recording just by using cat /dev/dsp.wav >recording.wav. To control virtual_oss at runtime you can use virtual_oss_ctl -f /dev/vdsp.ctl. The previous example creates one .ctl device file per virtual_oss configuration. Second configuration is a dummy and it uses /dev/null as a hardware device to achieve most of what dsp config is doing, only simpler. It creates only one device and that is /dev/vdsp.dummy which is stereo only. It uses the same resampling (-S), real-time priority (-i 8), sample rate (-r 48000), bit rate (-b 32) and buffer size (-s 768) as dsp config while creating similar .wav and .ctl devices.

Now you can use /dev/dsp and /dev/vdsp inside or outside of jail to have either a real hardware device or purely virtual one. All of those flags can be changed at runtime with virtual_oss_cmd. You can switch to “studio mode” as I call it (smaller buffer size), do your recording/production and then switch to “desktop mode” with larger buffer. Do note that virtual_oss can change buffer size on start as most applications using OSS API will read desired buffer size once, on initialization. Almost all flags supported by virtual_oss are also changeable during runtime via virtual_oss_cmd. Developer or not, all people sometimes need to switch their default input/output device so it is really handy in everyday use, too. Now if you need ALSA or SNDIO development inside that same jail, it’s just a matter of installing ports/packages and altering configuration to your liking (default sound device, for example).

Let’s see what studio means in technical terms. Core of every studio is a mixer and today’s mixers are usually also USB audio interfaces. For live performance and monitoring, mixer is where it all happens. For recording and (post)processing, the computer is doing it all. The description of the mixer and operating one is dependent on the model so I won’t cover it here, but the recording part is where FreeBSD shines.

Besides having architecture that allows for real time threads and less jitters (to be honest, using it for years I never saw any jitters at all!), it is really nice having storage feature like ZFS to make sure your recordings are safe. As with a big number of channels user space mixing and resampling is more efficient then the one in the kernel so virtual_oss and OSS provide an ideal combination. The core of every recording system is Digital Audio Workstation, or DAW for short. In my studio, the DAW of choice is Ardour. It is mature and stable and has great integration with JACK. For the set of effects I use Calf, Invada and EQ10Q plugins as they provide good implementations of reverbs, choruses, equalizers and flangers. My choice of drum sampler is Drumgizmo, but it’s no wonder being a contributor to the project and port maintainer. Drumgizmo is unique in a way that it records and plays samples (drum hits). The idea is to replicate a studio recording as close as possible, so its principle is to have, for example, a snare drum recorded with all 16 microphones. Although the microphone in the kick drum will record the snare hit very faintly, having all 16 microphones record every hit makes the recording sound like a live studio. Recording with a non-main microphone is called bleeding and is controllable in Drumgizmo. Bleed and humanizer (randomizing hit strength, timing and position on the pad/cymbal) make recording sound very natural. I personally use hardware guitar/bass/vocal processor and synth, so I’m not experienced with software alternatives, but I do know some of my friends really like Geon Kick and Yoshimi for electronic music and Guitarix for guitar based music.

For me the hardware mixer is doing most of the audio routing and monitoring, but for USB audio interfaces that are not stand-alone, software has to do it. Usually, one would do routing based on JACK and it is a valid option, but with virtual_oss there’s another one. Let me give you an example. In the past I used USB audio interfaces that are not stand-alone and what I constantly have as a requirement that the first input (my guitar) is routed to outputs 9 and 10 (my guitar processor). To achieve this, you can add “-M i,0,8,0,0,0 -M i,0,9,0,0,0” to virtual_oss options. Note that channel numbers start with zero. There are also interfaces that have separate main output for speakers and headphones. For monitoring it is not so great so if you’d like to mirror everything that goes to speakers (output 1 and 2) to headphones (outputs 7 and 8), you can use “-M o,0,6,0,0,0 -M o,1,7,0,0,0”. The options virtual_oss alone supports give you the ability to have EQ, compressor, loop back, HTTP streaming and more, but it would be too much for this article to describe it all. Man page contains all options and examples of how you can achieve different setups.

If you think there’s not much that FreeBSD brings to the table in the audio world, you’re right and wrong at the same time. What the operating system can do in terms of audio is provide real time support, efficient resampling and good choice of open source DAWs and plugins. But that is true only if you use that computer exclusively for music.

I believe that general purpose operating systems must be general enough to be the choice for any task and that is where FreeBSD shines: with jails, firewalls, virtual switches, ZFS, packages that are up to date, security updates and all the person would expect from an operating system. FreeBSD provides it all while being great for music, so with it power is literally under your fingertips.

If you think about it, MacOS and Windows are mostly desktop operating systems, Linux lacking proper ZFS support/integration hardly makes it good for storage, Solaris is too huge for a router and other BSDs probably don’t have the number of audio ports that FreeBSD has. So to put it short, FreeBSD shines at not needing anything particular: no special care needed wherever you run it and whatever combination of apps you choose.

Call me stubborn, but having one operating system on a router, server, desktop, laptop and RPi is a big deal for me, especially if it solves all my problems on any hardware I put it on. For reference, my desktop machine in the studio is a 12 year old i5 PC with 8GB of RAM which I also use for Python/React/C/C++ development and most of that development is in jails.

What I’m trying to say is that FreeBSD gives you means to run literally everything on one machine while not sacrificing any efficiency.

So to put it really short, FreeBSD is great for audio studio not because it brings some unseen features, but because it does not collide with anything on the system while providing real time support, so the feeling is “this is just a normal desktop with audio apps”. Maybe it doesn’t sound too good, but just having “normal desktop” and “real time support” in one operating system is far from usual, and FreeBSD might be the only operating system not explicitly built for real time but able to provide that.

I have to say, the best quality of FreeBSD is our community. It is so easy to get to the right answer with the mailing list like multimedia@ and #freebsd channel on libera.chat. My personal experience is that the FreeBSD community is open and approachable. There are no distro-specific questions, the handbook is for FreeBSD and not just one of the FreeBSD forks/distributions. Developers are approachable by people who often are not sure what they need to ask in the first place. In one word, getting the proper information feels really easy and proper audio setup in a studio is a breeze using this operating system.

Note: when using jack configure it to use the real hardware. That means /dev/dsp1 that is OSS device (in my case with 18 channels) instead of /dev/dsp which is virtual_oss device (in my case with 2 channels).

The work from Ka Ho Ng is in base and jack2 port is available (bugzilla 251125) mostly thanks to Florian Walpen. My review (D30149) added a simple example of OSS development and is just the first of OSS examples to follow. One oftenly requested improvement to the handbook is the section of sound and audio. Alfonso Siciliano has great experience with mixer API and has contributed mixertui port and has offered help in understanding that code for future documentation purposes. Maolan is DAW (digital audio workstation) which is FreeBSD specific for now that I’m writing as an attempt to learn DSP and MIDI development and is the code where most of my experience/documentation comes from. While I do appreciate software like jack, I strongly believe that FreeBSD should have a DAW with native API supported out of the box, so I’m hoping that Maolan will improve FreeBSD based studios once it’s at least beta. Also, making APIs nicer to work with and documentation/examples better is what we as FreeBSD community must do if we want developers on other operating systems to be more portable and FreeBSD friendly. Over the past few years the community showed more interest in that area than I could ever imagine, so I am really grateful for being able to use a 12 year old desktop for everything including studio recording, web conferencing and development with nothing but FreeBSD and ports/packages it provides and no extra repositories.

Ne mogu vise da ćutim

Sun, 02 May 2021 16:30:00 +0000

Ne pišem na srpskom iz prostog razloga: želim da me što više ljudi razume. Međutim, ono što me tišti su neki ljudi odavde i ako ikada nabasaju na ovaj post želim da ga dobro razumeju. Pošto nemam plan pisanja, možda će biti o jednoj, možda o više osoba, ali svakako želim da budem iskren sa sobom i sa drugima šta ja o tim osobama mislim i zašto to mislim. Razlog zašto pišem tek sada posle godina problema sa tim osobama je što sam mislio da je OK oprostiti, ali psiha me je dovela do grčenja mišića (da budem precizan, grče mi se listovi) te ne vidim drugi izlaz nego da to pustim iz sebe. Pa da počnem …

Skoro sve osobe o kojima bih ovde želeo da pišem su profesori. To je tužno. Samo kad pomislim koga mi kao društvo puštamo da se šeta među decom/omladinom je bolesno, u najmanju ruku. Ali sada ne želim da pričam o društvu, želim da pričam o jednoj specifičnoj osobi. Ta osoba je Mirjana Ivanović. Ukratko, žena je glupa i bezobrazna i izdržati fakultet sa njom je bilo katastrofalno. Ako obratite pažnju na temu njenog magistarskog (što je ujedno i predmet koji je predavala i to očajno) je konstrukcija kompajlera. Postoje 2 razloga zašto mi nije ogadila taj predmet: skripta iz koje smo učili nije njena i ja sam bio očaran time kako kompajler radi. Za one koji nisu toliko sreće imali da ih kompajleri toliko privlače, Mira je bez sumnje ubila makar i tračak nade u tim ljudima da ona priča nešto pametno. Iz tog predmeta mi je dala 9 jer “nije to to”. Zapravo, pre nego što mi je ponudila 9 bilo je par sranja koje je napravila. Prvo mi je postavila pitanje koje ne postoji u skripti (iz koje ona predaje i podsećam, na predmetu koji je magistrirala). Drugo, dok sam pričao na tabli šta znam, u jednom momentu sam stao jer je pričala sa kolegom koji je bio na tabli pre mene. Kako sam pomislio da me ne čuje i ne prati, odlučio sam da sačekam, na šta je ona rekla “žene mogu da rade 3 stvari u isto vreme, a ja evo slušam vas, slušam kolegu i upisujem mu ocenu”. Koje proseravanje, nisam mogao da poverujem, jer ako neko priča nešto na pola metra od vas, a neko drugi na 5 metara, vi onog drugog ne čujete zbog decibela ne zbog intelekta. Ne znam šta mi je bilo, ali ja sam u tom momentu počeo da se smejem ko budala, jer taj komentar drugo ništa nije ni zaslužio. Kad me je pitala šta je smešno rekao sam da njen muž kad nam predaje neku baš tešku lekciju kaže “ova je baš muška”. Možda ćete pomisliti da nisam baš morao tako da reagujem, ali em sam rekao apslutnu istinu, em je ta moja izjava usledila nakon jedno 6 godina njenog izdrkavanja. Te naredi asistentu da poobara studente tako da nema više od 5oro na ispitu, te moram da se raspravljam sa njom da “jne” u asembleru znači “jump if not equal” jer kako je ona objasnila, “jne” je suprotno od onoga što jeste, te imamo ceo semestar predavanje o Modula-2 (jebem ti kompajler i koji izabraše, tuljani glupi) na kome saznamo da ništa od toga neće biti na ispitu, nego da ćemo odrađivati neke časove (odrađivanje na fakultetu, eeeee bre!) na kojima će biti zadaci za ispit. Tih časova je bilo 2. Slovima: dva!!! Onda nam je objašnjavala kako treba čovek da se požrtvuje za firmu i ako je tvom detetu rođendan da treba da ga iskuliraš ako firmi treba prekovremeno. Ne samo to, nego je godinu dana pre toga pričala kako sa profesorom koji piše raspored su se ona i muž dogovorili da je raspored takav da uvek jedan od roditelja može biti sa decom. Ljudi su pisali radove koje je ona kao zadnja kučka predavala kao svoje ne potpisavši nikog od tih ljudi. To je jedna glupa, licemerna, nadasve bezobrazna i plitka osoba. Sada kada mi ne predaješ više ništa, Miro majku i oca krvave ti jebem za ona obaranja “fali ti zarez”. Želim ti da ti zafali deci na svakom koraku uz nečije debilno objašnjenje “zato što ja tako kažem”.

Drugi po redu je Đura Paunić. Kako je premator, valjda je odjebao više sa fakulteta pa ga nešto i nema po sajtu previše. Zapravo ima nekih pomena njegovih govana, kao što je knjiga prepuna grešaka koju nikad u svojoj karijeri nije ispravio ne bi li ljudi bili primorani da dolaze na predavanja da bi čuli ispravke njegovih brljotina. A predavanje go kurac: prepisuje iz knjige na tablu. Ispit koji se zove “Strukture podataka i algoritmi” iliti “najbitnije što jedan programer može znati” sam pisao na papiru i čak i kad sam ga položio nisam stigao da napišem sve što znam. Razlog za to je što su zadaci iz doba Tjuringa i ništa iz te knjige se ne koristi više na taj način a student treba u svakom momentu da drži u glavi 9 pokazivača i da zna gde pokazuju. Ako ste ikad išta programirali, znate da je to nemoguće. Naravno, kao i smrdljiva Mira, Đura je imao praksu “nemoj da mi je previše studenata prošlo”. Eksplicitno sam pitao njegove asistente da li je on ikada programirao i svi do jednog su rekli “apsolutno ne”. To znači da je matora izdrkotina pričala neke budalaštine koje smo mi trebali da naučimo napamet. Pride, još me je njegov asistent, Pešović, čiji link isto ne mogu da nađem, oborio na “nije kao u knjizi”. Upljuvah se ja da objasnim kako je moj algoritam i brži i manje memorije zauzima i daje tačan rezultat. Na kraju sam ga oterao u pičku materinu i kad me je pitao šta sam rekao “to što si čuo”. Kontao sam u tom trenutku a evo i 20 godina kasnije kako ja kod tog debilka ionako nikad neću položiti, pa što da mu se ne najebem majke. Kada sam pao i drugi put taj ispit koji jebeno znam, rekao sam sebi da ako ne položim treći put, idem sa nožem na fakultet, stavljam ga Đuri pod grkljan i ako ne napiše ceo ispit za 3 sata, svinja će da bude preklana! Kako nisam u zatvoru, očigledno je da sam iz treće položio taj ispit. Đura je govno koje ni jednom studentu informatike neće nedostajati. Ako pak nađete takvog studenta, znate da ste našli isto govno kao i on sam što je, jer ko još može imati dobro mišljenje o profesoru koji ne zna da programira i predaje najbitniji programerski predmet osim jebeni licemer i dupelizac.

Ovo su samo neki od profesora koji su mi zadavali noćne more godinama i da želim da se zna kakvi su. Danas da im ide auto u sustret da ih pogazi, skoro pa sam siguran da bih ćutao! Nikada ne bih ubio nekog, ali ne mogu da vidim situaciju u kojoj bih ovima nešto pomogao ikada.

FreeBSD Upgrade Bootloader

Fri, 16 Apr 2021 14:23:00 +0000

WARNING: Not tested on dual boot machines and probably doesn’t work

The recommended partitioning layout is to support BIOS and UEFI. The following is GPT partitioning (could be MBR in this simple example). Notice that the first two partitions are EFI and BIOS boot partitions. The EFI partition is nothing more than a FAT partition in most cases.

gpart show
=>       40  500118112  ada0  GPT  (238G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528    4194304     3  freebsd-swap  (2.0G)
    4728832  495388672     4  freebsd-zfs  (236G)
  500117504        648        - free -  (324K)

To see current UEFI settings like which disk/partition/file is configured for booting, run the following:

efibootmgr -v
BootCurrent: 0019
Timeout    : 0 seconds
BootOrder  : 0019, 000A, 000C, 0006, 0007, 0008, 0009, 000B, 000D, 000E, 000F, 0010, 0011, 0012, 0013
+Boot0019* FreeBSD HD(1,GPT,0a7e1ccc-8826-11eb-b711-f0def164c22a,0x28,0x82000)/File(\efi\freebsd\loader.efi)
                      ada0p1:/efi/freebsd/loader.efi (null)
# A LOT OF LINES REMOVED

Write disk (pmbr) and second partition (gptzfsboot) boot codes. If you’re using UFS instead of ZFS, change gptzfsboot to gptboot.

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
# if not mounted, mount efi partition under /boot/efi
# in my case, that's adaop1, as efibootmgr reported
cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi

Serbia vs Internet

Wed, 09 Dec 2020 12:20:00 +0000

Dear reader,

I really wish that you’re somewhere nice and you read this story with “in a land far away” at the beginning, wondering how strange we are. I really wish that place exists, because if we’re all and I mean ALL dealing with this, we screwed up as a global IT community.

This story starts 2.5 weeks ago when we decided to switch the Internet package we use. New one has 2 times download and 4 times upload compared to the old one, and why not. Also, we decided to take static IP so I don’t have to juggle dynamic DNS. It’s Monday, and we have to go to their office. It’s corona time, and we have to GO TO THE OFFICE!!! It’s even worse. We had to wait outside, it’s almost December at that time and I’m on northern hemisphere, so it’s fucking cold! We knew that it takes forever for any Serbian ISP to do anything, so my wife called, first, explaining what we want to do, so they wrote it down and said “just go to our office, show them your ID and it’s all done once you sign”. Ha! We fell for that again!? But yeah, it was shorter than without the preparation. The system didn’t work well, so it took eons to finish those papers. At the end the clerk said “I’ll send you an SMS once it’s done, so you don’t have to wait”. OK, so it took half eternity to do that and a few hours later we had a faster Internet. The fact is that with MTS you never know the speed of the Internet you’re signing for, because there is one speed declared on their site, one on their pamphlet and one printed out in their office. Of course, I’m talking about the same package! But OK, after speedtest you know what you have, and it’s 400/150MBit/s.

Now that was the easy part. Yeah, freezing my ass off was the easy part! Until tomorrow morning we didn’t get static IP no matter what I reset or reboot. I wrote them on the chat on their site and 30 minutes later, I have no Internet connection. The IP can not get more static than this, that’s for sure! I have no idea how many times we called, wrote to them, cried on twitter and whatnot. First support call came on Friday! I mean, by Friday you can have a heart operation, and it’s 4 days that it takes for MTS to even call (and ask for the imposibilionth time “what are the symptoms?”). Naturally, I didn’t wait for them to do something, I had better things to do. First, I noticed that when I restart their modem with factory settings, I can ping an IP for a short period after it boots. OK, so it means it connects, I have some connection, it fetches config from ISP server or wherever, loads it and all connections die. Now I know it’s not something physically broken, which would take a hell to freeze over before some ISP’s operator decided to go outside. Now all I need is admin user/pass to fix the modem config. Luckily, most providers here don’t practice security, so admin pass is all over the Internet. As a security expert I should be advocating for this to change, but HELL NO! How am I going to fix their screw ups if they change all the passwords?

The Internet is back, and I notice their device can be configured in bridged mode. In the same WEB interface you have user/pass fields pre-filled, and you can see it’s PPPoE connection. Fine! Inspecting the pass element in the browser gave me what I thought is the password. Better looking at it, it had only numbers and letters A to F. Damn! I hoped it’s not hash, but here we are. One thousand calls later I’m talking to a guy who can actually give me a user/pass for PPPoE. In the meantime, waiting for them to find somebody who knows what I’m talking about I configured ppp.conf in FreeBSD. I know it’s a good config because the error I get is “wrong password”. Here is the part of that conversation:

me: I need a PPPoE username and password
him: Username is <username>
me: ... and?
him: That's it.
me: If I give you a username for Facebook, can you login?
me: There has to be more!
him: Well, write this down (and he dictates my static IP)
me: I already have that data, I need the password.
him: telekom/telekom on 192.168.1.1
me: That's your router and user/pass for it. I need a PPPoE password.
me: P-P-P-O-E!!!
him: I'll have to call you back.

him: Your password is <curse in Serbian>
me: I'm sure somebody was listening to my talks!

By the way, he didn’t provide the whole username, because it’s in the form of <user>@open.telekom.rs (or something similar), but I didn’t care, it’s written in the WEB interface of an ISP device. Bridge mode, here I come! But damn, my speed is 40/40MBit/s. Looking around I found net/mpd5 and it got me to 200/150. OK, that’s nice! I read somewhere that igb has problems with PPPoE (something about not using all card’s queues). Luckily I have APU1 to replace that APU2. With APU1 now I have full speed. Hell yeah! But it’s not perfect, as AES-NI support in hardware came with APU2, so all VPNs will be slow. As a remedy for that, I can forward port to a home server and have a VPN concentrator there, but it’s not perfect.

Now we come to the fun part. One more reason why we need static IP is because IPv6 is a myth: everybody’s talking about it, but it can not be spotted in the wild. You know how your ISP is all nice and sweet when they need to push new technology or they just started giving some service? That’s when you need to strike! That’s a rare chance to get to somebody technical really fast and then ask real questions. Needles to say, when MTS started offering fibre optics, they started sending sales personnel to persuade people to switch their ISP. I asked them for IPv6 and the sales guy didn’t know, of course. So in a few days he came with a technical guy, we sat down and I asked a lot of questions, but once I got to IPv6, he said he has no idea what that is, but he has a number of the guy who probably knows about it. He called and gave me the phone. Needles to say, the third guy in a row has no idea what I’m talking about, so the answer is probably “no IPv6 address for you”. There was literally nobody else to call and ask for IPv6.

No IPv6 means I can use HE tunnel to get IPv6 over IPv4, and at least start learning about the technology and stack and whatnot. Once we had static IP, I realized DMZ is not going to work for that tunnel, so I needed to set the ISP device as a bridge, hence the above hassle. The tunnel broker is really nice as it gives you exact commands to type in your terminal as root for every operating system there is, so it’s the next best thing to having an actual IPv6.

I have APU1 currently working, APU2 that gave me headache and APU4 laying on my table waiting for me to configure it and try if it gives me full Internet speed while being able to utilize hardware AES-NI. One of these days, I’ll be on the Internet like it’s a normal thing in the 21st century.

Freenit Framework

Sun, 04 Oct 2020 00:36:00 +0000

For few years I worked on a startkit to get me faster results, then I started teaching using it, and now it’s a framework. So let me tell you a story.

Backend is written in Python and uses Flask and SmoREST, while frontend is composed of React, Material-UI and Axios. What I tried to achieve is “deployment easy / development portable” rule. Yeah, it’s quite some rule, and I have to tell you, it was not always easy. The nice thing that came out of it is that devops repo for Freenit has support for “plain old” POSIX compliant scripts to initialize a project and start development. Those scripts are also used in Docker, CBSD/Reggae jail and Vagrant, which are all preconfigured. As I use FreeBSD, I made Freenit port/package available, and as I use it in production I always keep the version updated. I also have a config of uWSGI that is automatically disabled in development, enabled in production.

This year we streamed course on development using Freenit, and to make introduction to development easier, I created designer and I started working on frontend documentation. Backend is kinda self-documented through Swagger and nice patterns that SmoREST/Marshmallow/Webargs enable. The “big thing” about designer is that it’s a drag n drop web design solution which uses JSON to save the work, but gives developer an opportunity to export it to React code with theme, styles and HTML/Material-UI components. The motive for this software is that I hate the fact that WEB designer and developer have to create the look from scratch, just using different tools/languages. This way, code is generated from design, and while it is not perfect, it is intentionally done so. What I mean is that the goal was to have saved and export file contain everything while being just json or js file. Naturally, I expect frontend developer to tidy up the code, split it into multiple files and so on. Other than that, generated code is decent, looked from developer’s perspective.

The course we streamed was about:

As this was the first year we streamed the course, we decided to do it in our native, Serbian, but next batch of courses will be in English, as we already have plans to record more.

One of my plans was to have everything upgradable with “pkg upgrade” if possible. As I’m developer of Freenit and maintainer of it’s port in FreeBSD, I know those are in sync. As a matter of fact, I know quite a lot of Python packages I use are in sync with FreeBSD ports. Most of that is due to responsive maintainers of those ports. I did some of that porting and the following picture is a tree of ports (represented by their bug ID) I had to port to have Freenit in ports.

For development of the Freenit backend I use pip, but for development of any product/service based on it I use FreeBSD’s pkg. The reason is that I want to fail fast and discover changes in libraries ideally the day they are published to pip, but use more stable packages for production. To get there, I had great help from the FreeBSD community creating that port, especially from Koobs. Although he does not agree on using FreeBSD’s pkg for Python packages in production, we both understand why the other guy does things the way he does. I really like the freedom the combination of this OS and language provides as you can really find what “best way to hosting” is for you and your needs.

Primarily, I’m system administrator. I do know C/C++/Python/JavaScript to a good degree, but I most enjoy doing sys admin stuff. What I really like about it is automation, and you might notice it in my coding as well. For example, I created UltiSnips snippets for most commonly used Freenit (and other) constructs, so I can have a page done with designer, exported to React, added backend integration through snippet, write backend model/endpoint/migration with snippets or tools and developed inside CBSD/Reggae, which is also used to publish the code. One feature I’m currently working on is a uniform setup for Python jails so that deployment based on Freenit can be unified, too. What you need is just USE_FREENIT=YES in your project/backend and you magically get build and publish functionalities.

Security is a long topic to add to already lengthy post. Also, it deserves a post on it’s own as it’s one of the really tricky topics in development of any kind. Needles to say, I have an opinion how it should be done in Freenit, so stay tuned for a new one!

OpenLDAP Multimaster

Wed, 25 Mar 2020 23:41:00 +0000

One thing I can tell you about email servers is that I’m really dumb to set it up properly. I’m mail admin since 2006 and 14 years later I still don’t know how to do it. I mean, yeah, I do run a mail server and it does work, but it’s far from satisfying, but that’s not what I want to talk about in this post. I want to talk about one part of email server: OpenLDAP.

You can think of OpenLDAP, or just ldap for short, as a lightweight database for users and groups. The reason I chose ldap over SQL is that it’s less resource hungry while being really flexible. One drawback is that it’s complicated as hell. Not the software or configuration itself, but errors are usually misleading (at least to me). On top of that, although I knew OpenLDAP supports N-way multimaster, I never found any decent documentation on how to actually configure a cluster. So in short, this is the configuration that works on my server:

ServerID        3 "ldap://ldap3.domain.tld"
moduleload      syncprov
overlay         syncprov
syncprov-checkpoint     10 1
syncprov-sessionlog     100
syncrepl        rid=31
                provider="ldap://ldap1.domain.tld"
                type=refreshAndPersist
                schemachecking=on
                retry="5 10 30 +"
                searchbase="dc=ldap"
                bindmethod=simple
                binddn="cn=root,dc=ldap"
                credentials="verysecret"
                starttls=yes
                tls_cacert=/etc/ssl/cert.pem
                tls_cert=/usr/local/etc/openldap/certs/fullchain.pem
                tls_key=/usr/local/etc/openldap/certs/privkey.pem
syncrepl        rid=32
                provider="ldap://ldap2.domain.tld"
                type=refreshAndPersist
                schemachecking=on
                retry="5 10 30 +"
                searchbase="dc=ldap"
                bindmethod=simple
                binddn="cn=root,dc=ldap"
                credentials="verysecret"
                starttls=yes
                tls_cacert=/etc/ssl/cert.pem
                tls_cert=/usr/local/etc/openldap/certs/fullchain.pem
                tls_key=/usr/local/etc/openldap/certs/privkey.pem
MirrorMode on

Of course, it is in FreeBSD jail and it uses letsencrypt certificates. There are few things you should note about above config. First, there are 3 ldap servers which are all masters. Second, ServerID, ldap URL and rid are somewhat connected: they all contain number 3 in them. That’s a convention I find easiest to follow and understand, and makes some errors somewhat easy to catch. For example, rid should never contain two same digits, like 33. Although ldap server itself won’t stop you, it’s easier this way as rid=33 means that server 3 should connect to itself, which is not good. You can have as much servers as you want and number of syncrepl sections in your configuration should be one less than the number of servers. FreeBSD slapd servie should be configured like this:

slapd_enable="YES"
slapd_flags="-u ldap -g ldap -h ldap://ldap3.domain.tld"

One thing you should be careful about is that ldap3.domain.tld must be resolvable. On top of that, it should resolve to the IP of the jail it’s running in. This is usually not the case as you probably point domain names to server IP, not jail IP. The way I solved it is with the little help of Unbound. As CBSD/Reggae already uses unbound, I created a fake auth zone for ldap3.domain.tld:

ldap3.domain.tld. SOA ldap3.domain.tld. hostmaster.ldap3.domain.tld. (
                  1998092901  ; Serial number
                  60          ; Refresh
                  1800        ; Retry
                  3600        ; Expire
                  1728 )      ; Minimum TTL
ldap3.domain.tld.            NS      ldap3.domain.tld.

$ORIGIN ldap3.domain.tld
@    A   1.1.1.1

Of course, you should replace 1.1.1.1 with the actual IP address of jail where ldap is running. This is not ideal, but if I ever find better solution I will certainly write about it. There is just one more thing you should worry about and that’s renewing certificates. As uid/gid of cert files is probably not the same as those running slapd service, there’s a little script I wrote that is executed every time I run letsencrypt client (dehydrated, in my case, ran once a week):

#!/bin/sh

DOMAIN="$1"
if [ -z "${DOMAIN}" ]; then
  echo "Usage $0 <domain>" >&2
  exit 1
fi

PRIVKEY=/usr/local/etc/openldap/certs/privkey.pem
CERT_DIFF="dummy"

if [ -e ${PRIVKEY} ]; then
  CERT_DIFF=`diff /etc/certs/${DOMAIN}/privkey.pem ${PRIVKEY}`
fi


if [ ! -z "${CERT_DIFF}" ]; then
  cat /etc/certs/${DOMAIN}/privkey.pem >/usr/local/etc/openldap/certs/privkey.pem
  cat /etc/certs/${DOMAIN}/fullchain.pem >/usr/local/etc/openldap/certs/fullchain.pem
  chown ldap:ldap /usr/local/etc/openldap/certs/*.pem
  chmod 600 /usr/local/etc/openldap/certs/*.pem
  service slapd restart
fi
exit 0

It should be ran as update_certs.sh domain.tld. You might not have letsencrypt certs in /etc/certs, so edit that script to conform to your paths and configuration.

NOTE: There are some blog posts that state you should use chain.pem for tls_cacert. That does not work. If you have trouble with your service, try running it as this:

/usr/local/libexec/slapd -u ldap -g ldap -h ldap://ldap3.domain.tld -d 1

It will run slapd in the foreground and spew a lot of messages to your terminal. Some of them might be helpful. Also, you might want to use other number than 1 for -d argument, but I found it’s the best verbosity level for me.

OpenLDAP has alternate configuration syntax usually called cn=config for short. It allows you to keep configuration in ldap itself and changing those values makes them active right away. To be honest, I perfectly understand why some data centers would want not to restart the service when they change configuration, but for my little server, that’s an overkill. Also, cn=config variables for multimaster are somewhat similar to those I showed here, so it should be almost easy to convert them. Also, official documentation for multimaster uses cn=config, so give it a try if you’re using cn=config.

FreeBSD Crossbow

Tue, 18 Feb 2020 17:48:00 +0000

Once I started learning about containers and surrounding technologies, I heard about Solaris Zones and Crossbow. In short, zones are containers, like jails, and crossbow is something like a vnet. They say it’s way more flexible and powerful. One of the things it can do is prevent you from setting up static IP. I thought that’s pretty important thing and I wanted to have that. With CBSD/Reggae I’m a little closer to saying “we have it”. Reggae sets up a jail named cbsd and inside it /dev/pf and DHCP server are configured in a very special way. Because DHCP process is running as dhcp user, and that user can not run pfctl, unless /dev/pf owning group is the same as DHCP process group. Luckily, devfs.rules allows one set of rules for host and other set for the jail. That means /dev/pf on host is owned by root:root while it’s owned by root:unbound inside the jail and mode is 660. Of course, DHCP runs under group unbound. The reason is that unbound files from host are nullfs mounted inside the jail. That way DHCP can edit unbound zones and add leased addresses to PF table. On host, that PF table is used to configure NAT, so basically allowing jails to reach Internet.

Let’s face it. What I just described is nowhere near to Solaris crossbow, but it’s the closest I can get.

CBSD Base Upgrade

Mon, 18 Nov 2019 19:36:00 +0000

Upgrade in CBSD means the same as in FreeBSD: increse only in patch version.

cbsd baseupdate
service cbsd restart

For upgrade you need to stop the jail, set it to new base and start the jail.

cbsd jstop nginx
cbsd jset jname=nginx ver=12.1
cbsd jstart nginx
cbsd etcupdate jname=cbsd mode=update from=12.0 to=12.1 mode=diff

CBSD will ask you how do you want new base files to be fetched, and default is to download them. Other options include compiling from code and using host system files as new base. If you upgrade more then one jail, the first jstart will create base jail for the rest to use.

Once you’ve upgraded all your jails, it’s time to cleanup

cbsd removebase ver=12.0

FreeBSD Upgrade

Mon, 18 Nov 2019 19:12:00 +0000

Upgrade in FreeBSD world means having a host on version 12.0 and doing similar procedure like in FreeBSD update, but ending up with a major or minor version number incresed, not patch version number.

TL;DR

env PAGER=/bin/cat freebsd-update upgrade -r 12.1-RELEASE
freebsd-version -ku
bectl create 12.0.11
freebsd-update install
reboot
freebsd-update install
reboot
pkg upgrade
freebsd-update install
reboot

freebsd-update utility will tell you if there is anything to be fetched. If there isn’t, just ignore the rest of the commands.

First, using freebsd-version and bectl you create a boot environment for the current version of FreeBSD. Then, first install will update only kernel. As FreeBSD kernels are backward compatible, your system can boot with newer kernel then the rest of the operating system. Second install will take care of FreeBSD base. If everything is OK, after another reboot, you should upgrade packages for the new OS version and run finall install which will take care of known package problems. The last reboot is there to ensure everything is working OK, as you might have some kernel modules, like drm-kmod, which are changed during the upgrade.

FreeBSD Update

Tue, 25 Jun 2019 13:03:00 +0000

As you might be comming from Linux background, like I am, you might not be familiar with the FreeBSD update procedure. The main difference is that FreeBSD has base system as one big component which is updated as a whole, plus it has packages like any other Unix.

TL;DR

env PAGER=/bin/cat freebsd-update fetch
freebsd-version -ku
bectl create 12.0.3
freebsd-update install
reboot

Check if there’s a known migration problem.

pkg upgrade
reboot

If you have jails, update them before last reboot.

freebsd-update utility will tell you if there is anything to be fetched. If there isn’t, just ignore the rest of the commands, but if there is, you probably want to know what is the current version using freebsd-version. The -k and -u options stand for kernel and userland, respectively. Those version can be different (only the patch level), so pick the higher one. In the example above, that’s 12.0-p3. If you’re running on ZFS, you can use bectl to create new boot environment out of the current one, so if update goes wrong, you can still boot system with the previous version. The boot environment is ZFS-only feature which allows for multiple root datasets which loader(8) knows how to boot into. In a sense, it’s like installing update and all the packages to new root partition, every time. If you’re not running your FreeBSD on ZFS, just skip this step. Finally, we install the actual update and reboot, so the new kernel and base system are loaded. As the final part, packages are updated.

If you have a machine you can not reboot for any reason and have access through VNC, serial console or other non-network channels, you can run this:

env PAGER=/bin/cat freebsd-update fetch
freebsd-version -ku
bectl create 12.0.3
shutdown now
freebsd-update install
exit
pkg upgrade

The shutdown command will not power your machine off. It will bring it into single user mode. In this mode, most processes are not running, only those to enable basic terminal functionallity (and a bit more, but let’s say nothing is running). It will ask for root password and once the actuall update is finished, exit will start the services like on fresh boot. The drawback is that kernel updates are not activated, but you still profit from the updates to the userland. As FreeBSD tries relly hard to maintain ABI compatibility on the patch level, this is safe enough to do, but it is advisable to reboot once after the update to load the new kernel.

Videos

Mon, 24 Jun 2019 18:31:00 +0000

DaFED #61 - Kako automatizacija procesa pomaže u razvoju sigurnijih aplikacija

FOSDEM 2023 - Reggae: cool way of managing jails/VMs on FreeBSD

EuroBSD 2022 - FreeBSD audio in amateur and professional setups

FOSDEM 2021 - Managing virtual resources with CBSD, and beyond

BSidesLjubljana 2019 - How Did We Teach (Ourselves) Security

FOSDEM 2019 - FreeBSD in Audio Studio

BSidesLjubljana 2018 - Modern And Secure Devops on FreeBSD

FOSDEM 2018 - CBSD Isolation Manager

BalCCon 2016 - FLOSS Way of Funding a Hackerspace

BalCCon 2015 - What is Hackerspace

IT Feels Like Constant Failure

Mon, 08 Apr 2019 20:10:00 +0000

Do you remember when any operating system just worked? Me neither, and I hate it. I mean, Windows was always just a joke. Just think about it, Microsoft said their git repository is 300GB in size. How do you compile, let alone read? One will probably think of Linux, but what about systemd and Richard Stallman? Honestly, I’m not sure what is more revolting: shitty system software, or the most annoying guy ever. One might think about BSDs, but what about the support? FreeBSD being most popular and getting most attraction still lacks support for a lot of hardware. And the hardware? I don’t remember when I bought a piece of hardware that didn’t totally suck. As a matter of fact, how do you even do proper programming when the underlaying hardware is fucked up? And where are we as IT going? We’re creating bunch of sites and software for what? I can’t remember how many times I’ve heard “we’re creating something revolutionary”. It’s like nobody can create just a regular software solving small problems. Why? I hate the current state, and the worst is the hardware industry. Just look at the spectra and meltdown. I’m ashamed and sorry for contributing to this whole circus, where it’s only important to sell, get revenue and bail out. I may never stop doing IT stuff, but I do hate it from the bottom of my soul. Also, recent years showed me that my colleagues are mostly not interested in the IT at all, but to get their salary and go home. I can’t blame them, but I can’t like them either. As a matter of fact, there are more and more programmers that have no education. I don’t mean formal, as I know some exceptional developers without a single day at the university, but wish to know more is rare. Like we’re playing this dumb game to put money into someone’s pocket, and that pocket ain’t ours. It’s like we’re doing everything we can to screw this industry that once was so beautiful. That’s why IT feels like a constant failure, because it is!

HardenedBSD Install Over Serial

Mon, 13 Aug 2018 00:45:00 +0000

If you have APU or similar router, it’s a big chance you’ll need serial console install procedure. You’ll have to mount install image before booting and change /boot/loader.conf so it includes the lines to use serial console:

boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole,vidconsole"

When the machine boots, it will ask you for prefered console type. Default (vt100) is just fine. The rest of the installation is just like on the normal machine, but you’ll have to modify /boot/loader.conf on the newely installed. Reboot, and boot off of USB key once again with the same procedure, but go into shell, instead of install.

mkdir /tmp/install
zpool import -R /tmp/install -f zroot
zfs mount zroot/ROOT/default

Now write the same lines for serial console to /tmp/install/boot/loader.conf and you should be set.

CBSD Reggae

Mon, 20 Nov 2017 03:00:00 +0000

If you didn’t get it by now, I’m huge CBSD fan. Actually, I’m CBSD developer now. :o) One of the things I play with is Reggae, and some people asked me to describe it and make a demo. So here goes the explanation why I created it, how to use it and what are my plans for it in the future.

As CBSD is used for almost everything in Reggae, I won’t discuss that part. So, besides jails, these are the topics I tried to cover with Reggae:

network and pf being static
environment for bhyve VMs
ansible provisioning
development envrionemnt

Let me explain in a bit more detail. If you use CBSD as Oleg (lead developer) likes it, it will do all the magic for you: configure bridge interface for VMs, dynamically change PF rules when jails go up and so on. While this is absolutely great feature, I know bunch of admins that would like to try CBSD but don’t like other software to alter their firewall rules (me included). To automate the process of CBSD initialization with PF rules that don’t change and interfaces that are fixed (except tap and epair), I created command reggae init. You can think of it as a test: it creates static config for CBSD network so you can test if everything is working as you would like it. As script grew, it became obvious to me that it can also be used to initialize the server, not just test weird CBSD config and such. Network interfaces are configured only because you have to NAT using interface names so those had to be static as much as possible. I like the setup where jails are on lo1 and virtual machines on bridge1. In /etc/pf.conf that’s

jail_if = "lo1"
bridge_if = "bridge1"
nat on $ext_if from { ($jail_if:network), ($bridge_if:network) } to any -> ($ext_if)

So that’s basically use case for the static network and how I tried to solve it. I’m sure there are better ways to do it, but I wanted to create proof of concept that we can discuss further.

As for VMs you need DHCP server to make it easier to work with, reggae init will also setup two jails: dhcp and resolver. DHCP will lease IPs to VMs and register them in DNS (resolver). What you’ll notice is that only jail on bridge1 is DHCP, and that’s because it has to be on the same bridge as the VMs. Also, DHCP server is ISC’s Kea which has control socket file you can use to get statistics and to reconfigure/reload Kea. I still didn’t poke that socket, but I hope that in the future it will provide enough stats that CBSD can use it internally. Having resolver working, jails registration was straight forward: master_poststart.d / master_prestop.d hooks that use nsupdate and RNDC key to add/delete jail’s IP to/from the zone. So now Reggae has two zones, my.domain and vm.my.domain, as IPs assigned to jails and VMs are from different IP ranges and I want to be clear is it a jail or VM just from looking at the name.

For Ansible to provision jails you have to use jail connection instead ssh. Also, as Ansible expects python binary to be installed in /usr/bin, you have to tell Ansible where python is on FreeBSD machines. So this is how the inventory of one jail looks like:

jail1 ansible_connection=jail ansible_python_interpreter='"/usr/bin/env python"'

To make it easier for me to run Ansible on a jail, I created some Makefiles to help me with common tasks. If you create hierarchy in your repo the way Reggae expects it, you can “just write Ansible playbook” and it will be applied to your jail with one make. I could use a shell script for that, too, but I wanted to leave room for parallelism in the future.

To be able to reach your jail with Ansible once it’s on the server, Reggae will create provision user, give it sudo priviledges and add public key to it’s ssh. What it does is cp ~/.ssh/id_rsa.pub <jail-data>/home/provision/.ssh/authorized_keys. It is a bit hardcoded, but it will get better support in the future. The idea is to use SSH’s ProxyCommand to make your host jump box to jail. As Ansible knows how to use jump box, you can provision or update your jails on the server.

To make all this possible, I created extra files like jail profile, skel, etc, but there is also script which switches your DNS entry in /etc/resolv.conf from the one DHCP provided (or you entered, if you use static IP) to the resolver (jail where BIND9 is running), there are hooks for (de)registering jails, … What I’m also trying to achieve is to have my dev environment be exactly the same (yeah, right, like that’s possible) as my production. Reggae also has development mode, in which it will mount your resository’s directory on host to /usr/src inside jail, create devel user with same UID:GID inside jail that user running make has on the host has and it adds one more target, so when you run make && make devel on yet uninitialized repository, it will provision it if needed, run /usr/src/bin/init.sh and /usr/src/bin/devel.sh, so by implementing those scripts you choose what happens on make devel.

Here are some example repos using Reggae:

Tilda Center website (development mode example)
EMail service (Ansible example)

What I’d like to see in the future is much of this work merged into CBSD on one way or another. This is what makes my development experience nicer, and as such I’d like it to be less hackish at some parts. Sometimes, it’s hackish because of a CBSD bug, sometimes because I can’t think of a better solution at the time. As I use it for testing CBSD in “uncharted theritories”, I like being able to find bugs that wouldn’t pop up otherwise. Also, as I’d like to use CBSD in every situation I can think of, hence making dev env as close to production as possible, some extra tools or CBSD features are needed and Reggae is one of them, at least for me. The parts that proove to be useful to other people will probably go into CBSD or some more official CBSD repository.

This is how reggae init feels like:

You can use it for development:

Or you can use it for project like service grouping:

Peek inside service:

Getting Started with Digital Audio

Mon, 20 Nov 2017 03:00:00 +0000

It’s bad to start your post with an apology, but I must: this one won’t deal with FreeBSD as much as set some foundation for the posts to come.

The base for a good digital audio workstation, or DAW for short is audio interface. That is the piece of equipment that will do the most demanding task of converting from analog to digital and vice versa. As audio we hear is, after all, analog one must pick it’s audio interface a bit more carefully than the rest of the gear, so here are my tips on choosing a decent one:

find as much as you can about it’s ADC (analog to digital converter) and DAC (digital to analog converter)
match impedance (more on that later)
as high sample/bit rate as possible with internal mixer using more bits than ADC/DAC so it has room to handle clipping (common these days is 24/32 bits)
ability to be world clock master and slave (more on that later)

Impedance is a fancy word for electrical resistance. It has to do with the fact that resistance of a device is not the same in all circumstances. One of those situations where this really matters is equalizer: one band actually has lower resistance for a certain frequencies, and higher for others. All I’m trying to emphasize here is that when you here “impedance” you should think “resistance” and keep in mind that it’s dynamic.

One thing all electrical circuits like is when impedance of it’s output matches the one on the input of the next step. When I say “like”, I mean least amount of energy is wasted in transit from one circuit to another (read: you get more signal/noise ratio) and the least amount of distortion is introduced (unfortunately, every device adds some distortion). So, to have a perfect audio interface, choose the one that has mic, line and hi-z inputs. Mic input should have 48V option which is needed for condenser microphones (studio microphones). Line is what most devices use, like mp3 players, other sound cards and synths. Hi-z is just a fancy name for “guitar input”. What you should look for with hi-z is a active/passive switch. Active pickups have small amp inside them and need battery, so they are easy to recognize. Passive pickups are the ones without battery, and they have 3 to 9 times lower output than active ones (depending on the chosen pair of active/passive pickups).

As digital audio IO must operate at the precise same frequency across all devices, once you get guitar or vocal processor, you’ll need to sync your audio interface and processor. There are multiple ways for achieving that and it mostly depends on the way you’re going to connect the devices, but let me explain why it’s important. All digital devices use “the clock”. It’s what tells them “hey, it’s time for the next sample” among other things. That clock is usually quartz crystal which has a property of oscillating when electric current is introduced. When you have two devices with their own clocks, they have slight differences in frequencies which come from slight differences in crystals inside them. You might think “I don’t care about few milliseconds” of delay, but that’s not what’s in stake here. If digital device misses the clock beat, all audio can become gibberish and noise. This is solved by having devices that can use external clock as it’s own. Obviously, one of the devices must “export” it’s internal clock (acts as master) to other devices (slaves). S/PDIF and AES/EBU digital connections can also transmit the clock but you have to check your devices for such capabilities as not all can work this way. The safest option is to have World Clock on all of your devices, where your audio interface is the master. World Clock connector is BNC. It is especially important when you want to connect multiple devices to your computer (for example, via USB) as otherwise you’ll get a lot of small errors known as jitters.

For someone starting with audio, this must sound terribly boring, and maybe too technical, so please leave a comment if I didn’t cover something properly, or you have an idea how to make it more fun.

FreeBSD Cloud and DevOps 3

Sun, 01 Oct 2017 12:34:00 +0000

By now you know how to manage jails with Makefile. It’s nice, but I took it a bit further this past few months. I realized that Makefile can effectively replace Vagrant, so I created program called Reggae: REGister Globaly Access Everywhere. In short it consists of few scripts that (de)register jail in/from Consul. As Consul acts as DNS, too, that means that when your jail is up, other jails using Consul as DNS will know about it. Also, Consul can be used for other things, but it’s a different topic.

From the last blog post in this series you know how to use Makefile for these tasks, but I’ll run through some of the Makefiles from my project. There are 3 .mk files in Reggae:

project.mk
service.mk
ansible.mk

So let me explain how it works on the example of the first project I used it with: mail server. If you look at the mail as a project, it consists of few services like ldap, webmail, mail (dovecot + postfix) and so on. I have mail/Makefile which looks like this:

REGGAE_PATH = /usr/local/share/reggae
SERVICES = letsencrypt https://github.com/mekanix/jail-letsencrypt \
           ldap https://github.com/mekanix/jail-ldap \
           mail https://github.com/mekanix/jail-mail \
           jabber https://github.com/mekanix/jail-jabber \
           webmail https://github.com/mekanix/jail-webmail \
           web https://github.com/mekanix/jail-web \
           webconsul https://github.com/mekanix/jail-webconsul
DOMAIN=lust4trust.com

.include <${REGGAE_PATH}/mk/project.mk>

Yes, that’s the whole file! The core of the Reggae is SERVICES in project.mk, so let’s see how it deals with it:

up: fetch setup
.if defined(service)
	@echo "=== ${service} ==="
	@${MAKE} ${MAKEFLAGS} -C services/${service} up
.else
.for service url in ${SERVICES}
	@echo "=== ${service} ==="
	@${MAKE} ${MAKEFLAGS} -C services/${service} up
.endfor
.endif

Lets break it down. First, up target depends on fetch and setup. Once everything needed is downloaded and initialized, one of the two if branches will be triggered. You can run it with make up or make service=ldap up. Former runs up on all services (or jails in our case) and later get’s only ldap jail up. So the if is there to see if service=<something> is present on the command line. If it’s not, biggest problem for Reggae starts. That for loop is where I lost most time figuring out how to have something I would call “list of tuples” in Python. After a lot of experimenting, I realized that if I use service and url as indexes in the same loop, it will do what I want. With down target you have to do it in reverse, as some jails might depend on other jails (for nullfs mount, perheps?). As SERVICES is array, not array of pairs, you have to reverse the indexes, too: url and service in for loop.

down: setup
.if defined(service)
	@${MAKE} ${MAKEFLAGS} -C services/${service} down
.else
.for url service in ${SERVICES:[-1..1]}
	@${MAKE} ${MAKEFLAGS} -C services/${service} down
.endfor
.endif

Service uses all the same Makefile tricks, so let me just show how I provision the jails. I implemented ansible.mk as an example, but Reggae is not Ansible centric. First thing is to mark the default target to run:

.MAIN: up

This way it doesn’t matter which target is first, up will be triggered if you just type make. This also solves the problem of adding targets wherever you like thus extending what can be done with your project. So let’s look at how provisioning works.

provision:
	@touch .provisioned
.if target(do_provision)
	@${MAKE} ${MAKEFLAGS} do_provision
.endif

This is in service.mk in Reggae. If you defined do_provision or included ansible.mk from Reggae, provision will run it. As a matter of fact, this is how ldap service Makefile looks like:

SERVICE = ldap
REGGAE_PATH = /usr/local/share/reggae
CUSTOM_TEMPLATES = templates

.include <${REGGAE_PATH}/mk/ansible.mk>
.include <${REGGAE_PATH}/mk/service.mk>

Including ansible.mk before service.mk ensures that do_provision is defined when provision target from service.mk is parsed. Also, .MAIN will ensure that running just make doesn’t run the first target from ansible.mk.

If you need to mount something extra in your jail, you can define EXTRA_FSTAB with the value of path to fstab containing extra mounts. Also, in order for provision to work, some files had to be generated from templates, so this is the directory hierarchy you need in your service repo:

templates/site.yml.tpl
playbook/group_vars
playbook/inventory
playbook/roles

You should know what those are if you ever used Ansible. Also, playbook directories are the ones where Reggae will either generate some files (that should be in .gitignore) or expect other files to be.

The last piece is registering with Consul. So, this is how I configured my /etc/rc.conf.d/consul:

consul_enable="YES"
consul_args="-bind=127.0.2.1 -client=127.0.2.1 -recursor=8.8.8.8 -ui -server -bootstrap"

You can run Consul in jail, too. As a matter of fact, I do and it’s IP is special: 127.0.2.1. If you do that, Reggae will just work with Consul. In some of the future posts I’ll explain how you can use Ansible with Consul to provision your jails. If you’re inpatient, you can check out my mail project.

NuttX and Clang

Mon, 03 Jul 2017 15:23:00 +0000

I am proud owner of a home audio studio, but it has a big flaw: hardware is not open source. You may think it’s not a big deal, but once you want to alter something, you realize you’re stuck. Let me explain. My mixer is digital and you control it over some other device (computer, or android based device). It’s a nice feature, and it has USB connection, so it can act as an audio interface. The problem with that is 48kHz sampling, which is OK for live gigs, but not so much for studio recordings. On the other hand, I have audio interface with 96kHz sampling, but it can not work as stand alone mixer. To be honest, I don’t know anything about DSP and embeded programming, but I said to myself “I know I can do better than this”. That’s how “the ride” began.

Since then I really wanted to make Arduino Due working, but for some reason GCC on FreeBSD gave a faulty binary. Back then I was desperate and I knew there is absolutely no way I can fix the issue with the compiler. Hell, I couldn’t even use it as developer, let alone debug it. So there came Clang/LLVM.

In one of the previous posts I wrote about compiling blink LED test with Clang. That gave me hope that FreeBSD might be the perfect platform for me for embeded programming. As my end goal is to create a digital mixer with proper sampling rate, I knew from the begining I can not just write any code. It must be real time and optimized really good. I’ve got to be honest with you: there are two obstacles for me to write such a code. First one is that Arduino libraries are not optimal. I didn’t check, but I do have friends who are professional embeded programmers who told me that. Second one is that I’m system administrator. You can not imagine the desparation of a sys admin staring at the board which doesn’t blink.

Now for the good part. I discovered NuttX. A friend in Tilda hackerspace asked me how do I search for such cool things, and the only answer I had was “gut feeling”. Later I discovered that even Sony is using it for audio in some of their products. Even better, they use C++11, which is like a scripting language compared to “plain” C++ I used back in 2008. When I say “used” it’s an overstatement. I think more apropriate phrase would be “I played with it”. So, I made my goal to make NuttX compile with Clang, and I made it. You can check out my fork for now, until the patch makes it into the official repository. Although I used FreeBSD for development, I hope it’s generic enough to be used on other OSes which Clang/LLVM supports. It still lacks LLVM libc++ support, and that’s what we’ll be working on in the hackerspace today (Tuesday it’s embeded programming day) and in the future.

So, the final product will be digital mixer which can be controlled over network based on NuttX, Clang, Nucleo F401RE, and Cirus Logic DAC / ADC. The DAC/ADC choice was made by a new found friend from France I will talk about in some of the next posts (tramendesly interesting guy) who also wrote audio and midi server for NuttX.

FreeBSD and USB MIDI

Sat, 17 Jun 2017 23:37:00 +0000

The problem I was having was that I could record MIDI, but not send it to hardware output. For impatient, just run chmod o+rw /dev/umidi*. Yes, it was that stupid and it took me few days to realize what’s the problem, so let’s go through my setup and debug process. It will be fun, I promice.

I never used hardware MIDI before, although I used software MIDI for various things for years, so I have a rough feeling how things should work. Anyway, I have a footswitch (Behringer FCB1010) MIDI out into USB audio interface (Presonus AudioBox 1818VLS) in. Then jack_umidi -d /dev/umidi0.0 exposes hardware MIDI ports as jack MIDI ports. You can use jack_lsp to display all jack ports and jack_connect <out> <in> to connect whatever is coming from input to output and make your audio interface act as MIDI thru. That way I use Presonus as a “proxy” between FCB1010 and my guitar processor (Line 6 POD X3 Pro) in order to be able to record controls at the same time as I record dry guitar. I noticed that if I remove Presonus as proxy, controls work, but with it I couldn’t get it to work. I tested to make sure jack is emitting MIDI messages all, and it did. I tested hardware using my wife’s laptop and Linux, and that worked. At that point I realized it can be too many things, as I have too much apps in my setup, so I decided to make it as simple as possible: write a MIDI program in C based on synth example. To be precise, I wanted to use the open_midi_device and then whatever I read from it, I write to it back. The dumbest MIDI thru ever! Given example uses MIDI device in read only mode, but I needed read/write. Once I tried to alter it, I got Permission denied. Looking at /dev/umidi0.0 permissions, no wonder, because it’s owner is root, group is operator and it’s mod is 644. To make this right, add this rules to /etc/devfs.rules (create if it doesn’t exist):

[localrules=5]
add path 'umidi*' mode 0666

To make those rules active, add devfs_system_ruleset="localrules" to /etc/rc.conf. On next reboot everything will be just fine.

Have fun with the MIDI!

Arduino Due and Clang

Fri, 02 Jun 2017 04:00:00 +0000

Since October last year I’ve been trying to get my FreeBSD box compile working binary for Arduino Due. For some reason, GCC produced binary is missing a section which I observed through readelf. I tried to debug this, but I’m not even novice kernel developer, let alone bare metal embeded programmer. All that changed tonight (or should I say in the 4am today) when I stumbled upon just the right post. I want to be clear, Hanno Braun did all the work, I just want to point out how to get it working on your FreeBSD box.

I use -STABLE branch currently so system compiler is Clang 4.0. Beside that you’ll need few extra packages.

pkg install bossa arm-none-eabi-gcc

Bossa is used to upload the binary to Arduno and GNU linker for ARM is used as LLDB is not ready, yet (or I didn’t find example how to use it with Arduino). Also, you’ll need to add your user into dialer group.

pw group mod dialer -m <user>

Let’s say we have blink.c file with the code which blinks the onboard LED and linker-script.ld mentioned in the Hanno’s post. Just for reference, this is how he compiles the binary:

cc -nostdlib -ffreestanding -target arm-none-eabi -march=armv7-m -Tlinker-script.ld -Wl,--entry=start blink.c -o blink.elf
arm-none-eabi-objcopy -O binary blink.elf blink.bin

On my box, resulting blink.bin is 332 bytes! Not bad!

The flashing part is a bit different on FreeBSD. First, you have to issue soft erase by connecting to serial port provided by Arduino using 1200 baud. You just have to connect, so nothing extra is needed.

cu -l /dev/cuaU0 -s 1200

Also, you’ll have to tell Bossa utility which device file to use.

bossac --write --verify -p /dev/cuaU0 --boot -R blink.bin

Your Arduino Due should be blinking, now. Just for the sake of completeness, I’ll show what Hanno wrote in .c and .ld file.

blink.c:

// This is the top of the stack, as provided to us by the linker.
extern unsigned int _estack;


// This is a partial definition of the vector table. It only defines the
// first two entries which, as far as I can tell, are the minimum needed
// for a program to work at all.
// Space for the other interrupt handlers is reserved. I'm not sure if this
// is necessary, but I can imagine that the vector table not having the
// right length could cause all kinds of problems (imagine if it was too
// short, and the linker would place something else directly after it).
typedef struct {
	void *initial_stack_pointer_value;
	void *reset_handler;

	char other_interrupt_vectors[44 * 4]; // space for 44 32-bit pointers
} VectorTable;


void start();


// The vector table. We're using GCC-specific functionality to place this
// into the .vectors section, not where it would normally go (I suppose
// .rodata). The linker script makes sure that the .vectors section is at
// the right place.
__attribute__ ((section(".vectors")))
const VectorTable vector_table = {
	(void *)(&_estack),
	(void *)start,
};



// Addresses of several registers used to control parallel I/O.
static volatile int * const pb_pio_enable          = (int *)0x400E1000;
static volatile int * const pb_output_enable       = (int *)0x400E1010;
static volatile int * const pb_set_output_data     = (int *)0x400E1030;
static volatile int * const pb_clear_output_data   = (int *)0x400E1034;

// Bit mask for PB27. This is pin 13 (the built-in LED) on the Arduino Due.
static const int pb27_mask = 0x08000000;

// Addresses of several registers used to control the real-time timer.
static volatile int * const timer_mode_register  = (int *)0x400E1A30;
static volatile int * const timer_value_register = (int *)0x400E1A38;


// As the name suggests, this function sleeps for a given number of
// milliseconds. Our replacement for Arduino's delay function.
void sleep_ms(int milliseconds) {
	int sleep_until = *timer_value_register + milliseconds;
	while (*timer_value_register < sleep_until) {}
}

// This function is the entry point for our application and the handler
// function for the reset interrupt.
void start() {
	// Enable PB27 (pin 13) and configure it for output.
	*pb_pio_enable    = pb27_mask;
	*pb_output_enable = pb27_mask;

	// Set the timer to a resolution of a millisecond.
	*timer_mode_register = 0x00000020;

	// Continuously set and clear output on PB27 (pin 13). This blinks
	// the Due's built-in LED, which is the single purpose of this
	// program.
	while (1) {
		*pb_set_output_data = pb27_mask;
		sleep_ms(200);
		*pb_clear_output_data = pb27_mask;
		sleep_ms(200);
	}
}

linker-script.ld:

/* ----------------------------------------------------------------------------
 *         SAM Software Package License
 * ----------------------------------------------------------------------------
 * Copyright (c) 2012, Atmel Corporation
 *
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following condition is met:
 *
 * - Redistributions of source code must retain the above copyright notice,
 * this list of conditions and the disclaimer below.
 *
 * Atmel's name may not be used to endorse or promote products derived from
 * this software without specific prior written permission.
 *
 * DISCLAIMER: THIS SOFTWARE IS PROVIDED BY ATMEL "AS IS" AND ANY EXPRESS OR
 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT ARE
 * DISCLAIMED. IN NO EVENT SHALL ATMEL BE LIABLE FOR ANY DIRECT, INDIRECT,
 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
 * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
 * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 * ----------------------------------------------------------------------------
 */

OUTPUT_FORMAT("elf32-littlearm", "elf32-littlearm", "elf32-littlearm")
OUTPUT_ARCH(arm)
SEARCH_DIR(.)

/* Memory Spaces Definitions */
MEMORY
{
	rom (rx)    : ORIGIN = 0x00080000, LENGTH = 0x00080000 /* Flash, 512K */
	sram0 (rwx) : ORIGIN = 0x20000000, LENGTH = 0x00010000 /* sram0, 64K */
	sram1 (rwx) : ORIGIN = 0x20080000, LENGTH = 0x00008000 /* sram1, 32K */
	ram (rwx)   : ORIGIN = 0x20070000, LENGTH = 0x00018000 /* sram, 96K */
}

/* Section Definitions */
SECTIONS
{
    .text :
    {
        . = ALIGN(4);
        _sfixed = .;
        KEEP(*(.vectors .vectors.*))
        *(.text .text.* .gnu.linkonce.t.*)
        *(.glue_7t) *(.glue_7)
        *(.rodata .rodata* .gnu.linkonce.r.*)
        *(.ARM.extab* .gnu.linkonce.armextab.*)

        /* Support C constructors, and C destructors in both user code
           and the C library. This also provides support for C++ code. */
        . = ALIGN(4);
        KEEP(*(.init))
        . = ALIGN(4);
        __preinit_array_start = .;
        KEEP (*(.preinit_array))
        __preinit_array_end = .;

        . = ALIGN(4);
        __init_array_start = .;
        KEEP (*(SORT(.init_array.*)))
        KEEP (*(.init_array))
        __init_array_end = .;

        . = ALIGN(0x4);
        KEEP (*crtbegin.o(.ctors))
        KEEP (*(EXCLUDE_FILE (*crtend.o) .ctors))
        KEEP (*(SORT(.ctors.*)))
        KEEP (*crtend.o(.ctors))

        . = ALIGN(4);
        KEEP(*(.fini))

        . = ALIGN(4);
        __fini_array_start = .;
        KEEP (*(.fini_array))
        KEEP (*(SORT(.fini_array.*)))
        __fini_array_end = .;

        KEEP (*crtbegin.o(.dtors))
        KEEP (*(EXCLUDE_FILE (*crtend.o) .dtors))
        KEEP (*(SORT(.dtors.*)))
        KEEP (*crtend.o(.dtors))

        . = ALIGN(4);
        _efixed = .;            /* End of text section */
    } > rom

    /* .ARM.exidx is sorted, so has to go in its own output section.  */
    PROVIDE_HIDDEN (__exidx_start = .);
    .ARM.exidx :
    {
      *(.ARM.exidx* .gnu.linkonce.armexidx.*)
    } > rom
    PROVIDE_HIDDEN (__exidx_end = .);

    . = ALIGN(4);
    _etext = .;

    .relocate : AT (_etext)
    {
        . = ALIGN(4);
        _srelocate = .;
        *(.ramfunc .ramfunc.*);
        *(.data .data.*);
        . = ALIGN(4);
        _erelocate = .;
    } > ram

    /* .bss section which is used for uninitialized data */
    .bss ALIGN(4) (NOLOAD) :
    {
        . = ALIGN(4);
        _sbss = . ;
        _szero = .;
        *(.bss .bss.*)
        *(COMMON)
        . = ALIGN(4);
        _ebss = . ;
        _ezero = .;
    } > ram

    . = ALIGN(4);
    _end = . ;

    /* .stack_dummy section doesn't contains any symbols. It is only
       used for linker to calculate size of stack sections, and assign
       values to stack symbols later */
    .stack_dummy :
    {
        *(.stack*)
    } > ram

    /* Set stack top to end of ram, and stack limit move down by
     * size of stack_dummy section */
    __StackTop = ORIGIN(ram) + LENGTH(ram);
    __StackLimit = __StackTop - SIZEOF(.stack_dummy);
    PROVIDE(_sstack = __StackLimit);
    PROVIDE(_estack = __StackTop);
}

FreeBSD Cloud and DevOps 2

Tue, 04 Apr 2017 12:17:00 +0000

So you have all the pieces installed and configured. Let’s do something on top of that. I don’t like Vagrant, now that CBSD and Jails are around, but it’s perfect for DevOps jobs. I remember a friend of mine using Makefile for DevOps, even if under it is Docker or Vagrant. It’s small, portable and available on every OS worth mentioning. Nice way to keep everything low on resource usage.

So, let’s say we want to have at least 2 targets: up and setup. Setup would set up config files, where up would do the actual job. For example:

PROJECT=myproj

up: setup
	@sudo cbsd jstart ${PROJECT} || true

setup:
	@sed -e "s:PROJECT:${PROJECT}:g" provision/inventory.tpl >provision/inventory
	@sed -e "s:PROJECT:${PROJECT}:g" provision/group_vars/all.tpl >provision/group_vars/all
	@sed -e "s:PROJECT:${PROJECT}:g" provision/localhost.yml.tpl >provision/localhost.yml
	@sed -e "s:PROJECT:${PROJECT}:g" provision/site.yml.tpl >provision/site.yml
	@sudo cbsd jcreate jconf=${PWD}/cbsd.conf || true
	@sudo sh -c 'sed -e "s:PWD:${PWD}:g" -e "s:PROJECT:${PROJECT}:g" fstab.conf >/cbsd/jails-fstab/fstab.${PROJECT}'

You can see that up target is using cbsd.conf and fstab.conf. You can generate cbsd.conf with cbsd jconstruct-tui, like usual, just choose not to start it once you configure it and CBSD will save the config in temporary file you can use as a template. The fstab.conf is simple:

/cbsd/jails-data/PROJECT-data/etc /etc nullfs rw 0 0
/cbsd/jails-data/PROJECT-data/root /root nullfs rw 0 0
/cbsd/jails-data/PROJECT-data/tmp /tmp nullfs rw 0 0
/cbsd/jails-data/PROJECT-data/usr/home /usr/home nullfs rw 0 0
/cbsd/jails-data/PROJECT-data/usr/local /usr/local nullfs rw 0 0
/cbsd/jails-data/PROJECT-data/usr/compat /usr/compat nullfs rw 0 0
/cbsd/jails-data/PROJECT-data/var /var nullfs rw 0 0
PWD /usr/home/devel/workdir nullfs rw 0 0

One nice thing Vagrant has is it will provision your VM if it has provisioner configured if it’s the first time you’re starting that VM. Let’s add that:

up: setup
	@sudo cbsd jcreate jconf=${PWD}/cbsd.conf || true
.if !exists(.provisioned)
	@${MAKE} ${MAKEFLAGS} provision
.endif

provision:
	@sudo ansible-playbook -i provision/inventory provision/site.yml
	@touch .provisioned

Instead of vagrant ssh I added make login:

login: up
	@sudo cbsd jlogin ${PROJECT}

Finally, there are tear down targets, as well:

down: setup
	@sudo cbsd jstop ${PROJECT} || true
	@sudo ansible-playbook -i provision/inventory provision/teardown.yml

destroy: down
	@rm -f provision/inventory provision/site.yml provision/group_vars/all .provisioned
	@sudo cbsd jremove ${PROJECT}

All that’s left now is to write Ansible playbook, but I’ll leave that for some other post.

Sing, beastie, sing!

Wed, 25 Jan 2017 19:00:00 +0000

FreeBSD digital audio workstation, or DAW for short, is now possible. At this very moment it’s not user friendly that much, but you’ll manage. What I want to say is that I worked on porting some of the audio apps to FreeBSD, met some other people interested in porting audio stuff and became heavily involved with DrumGizmo - drum sampling engine. Let me start with the basic setup.

FreeBSD doesn’t have hard real-time support, but it’s pretty close. For the needs of audio, FreeBSD’s implementation of real-time is sufficient and, in my opinion, superior to the one you can get on Linux with RT path (which is ugly, not supported by distributions and breaks apps like VirtualBox). As default install of FreeBSD is concerned with real-time too much, we have to tweak sysctl a bit, so append this to your /etc/sysctl.conf:

kern.timecounter.alloweddeviation=0
hw.usb.uaudio.buffer_ms=2 # only on -STABLE for now
hw.snd.latency=0
kern.coredump=0

So let me go through the list. First item tells FreeBSD how many events it can aggregate (or wait for) before emitting them. The reason this is the default is because aggregating events saves power a bit, and currently more laptops are running FreeBSD than DAWs. Second one is the lowest possible buffer for USB audio driver. If you’re not using USB audio, this won’t change a thing. Third option is for defining default latency if program doesn’t set one. Every program can change this value for it self, but most of them don’t deal with this. Fourth one has nothing to do with real-time, but dealing with programs that consume ~3GB of RAM, dumping cores around made a problem on my machine. Besides, core dumps are only useful if you know how to debug the problem, or someone is willing to do that for you. I like to not generate those files by default, but if some app is constantly crashing, I enable dumps, run the app, crash it, and disable dumps again. I lost 30GB in under a minute by examining 10 different drumkits of DrumGizmo and all of them gave me 3GB of core file.

If you have audio interface with more than 8 channels, you’ll need virtual_oss and virtual_oss_ctl. The decision was made that more than 8 channels of audio are more suitable to be mixed, resampled and generally processed in user space. My rc script for virtual_oss is still pending, as I just can’t find the time to work on it. Copy virtual_oss from that repo to /usr/local/etc/rc.d and it will start virtual_oss assuming your audio interface has 18 channels. You have to add virtual_oss_enable="YES" to your /etc/rc.conf and you can alter the arguments by adding your own virtual_oss_flags="...". For example, take a look at the rc script, as it has default value of virtual_oss_flags.

Next is my jack setup, which is oneliner:

# jackd -r -d oss -r 88200 -C /dev/vdsp.jack -P /dev/vdsp.jack -i 18 -o 18

I’m not using real-time for jack, as it’s not supported for a non-root process to raise it’s priority to real-time. You can do that by sudo rtprio 10 -(pgrep jackd).

Currently, Ardour 5.5 and DrumGizmo are in conflict. Our hackerspace forked FreeBSD ports and added a quick patch for Ardour which removes the conflict. I talked to the maintainer of the port about it and he’s working on a proper patch and will try to push it upstream, to Ardour developers.

As there are some resampling problems with virtual_oss, you’re advised to use PulseAudio (not my favorite solution) by telling it to use virtual_oss. You’ll have to add the following to the /usr/local/etc/pulse/default.pa:

load-module module-oss device="/dev/vdsp.jack" sink_name=output source_name=input

And just for the reference, this is my virtual_oss config:

virtual_oss_enable="YES"
virtual_oss_flags="-S -i 8 -C 18 -c 18 -r 88200 -b 32 -s 384 -f /dev/dsp0 -c 2 -d dsp -c 18 -d vdsp.jack -t vdsp.ctl -M i,0,8,0,0,0 -M i,0,9,0,0,0"

-S: enable resampling
-i: enable real time
-c/-C: 18ch in/out
-r: sampling rate
-b: bits
-s: size of buffer
-d: virtual oss device to create
-t: virtual oss control device to create
-M: mirror first input to outputs 8/9

The reason I mirror first input (input zero) to 8/9 is because I use my first input for guitar, and outputs 8/9 are towards guitar processor. With this setup I can play OSS, JACK and PulseAudio sound all at the same time, which I was not able to do on Linux.

Now sing, beastie!

FreeBSD Cloud and DevOps 1

Sun, 08 Jan 2017 16:00:00 +0000

The reason I tried FreeBSD in August, after 11 years of being happy with Gentoo then Debian was the article about Docker on FreeBSD. I knew about PF, ZFS and DTrace from the past, but ability to run Linux images was just too good to be true. To the extent, it is too good, and the reason is that ABI Linux support is not complete. If you happen to need that missing bit in ABI, it’s a show stopper for anybody.

Poor was the result, I’m afraid, but there’s a positive side: Docker is not why I’m stuck with FreeBSD now. :o) There are so much better implementations of container technologies, I abandoned Docker completely. That and Docker’s continuous failure make you think of changing the tech stack completely. In this serie, I’ll walk you through my current stack for production and devops.

For start, let me give you some of the vocabulary:

ZFS - File system and volume manager.
Jail - Container technology, most similar to Linux LXC or Solaris Zones. Some think of it as chroot on steroids.
PF - Packet Filter - firewall.
BHyve - Type 2 hypervizor, or what provides VMs - virtual machines.
DNSMasq - DHCP and DNS server - keeps track of which virtual machine is where.
CBSD - Manages jails and virtual machines - create, up, down, destroy.
Sudo - Execute stuff as root when needed - devops side, mostly.
Make{,file} - Make is similar to GNU make with all it’s rules in Makefile.
Resolvconf - Software which configures /etc/resolv.conf

For this post I want you to get a working CBSD env. I’ll just assume you have machine which you’ll dedicate to FreeBSD only. Tips for install: use “Guided ZFS partitioning” and create user when asked. You’ll get to know the ZFS over time, but for now think of it as a requirement for easier jail and VM management. So, the only thing you have after install is FreeBSD base system on ZFS pool (by default, it’s name is zroot), root user and a regular user. As root prepare the CBSD:

# zfs create -o mountpoint=/cbsd zroot/cbsd
# pkg install cbsd dnsmasq bind-tools tightvnc
# env workdir="/cbsd" /usr/local/cbsd/sudoexec/initenv
# mkdir /usr/local/etc/dnsmasq.d

What this does is creates ZFS dataset which will be mounted on /cbsd as soon as zfs create finishes. For now you can think of a dataset as of a partition which can be created and destroyed while ZFS pool is mounted. Don’t worry if you don’t undestand ZFS features, it’s really complex piece of software.

After ZFS dataset creation, pkg installs CBSD and CBSD is initialized. I have few tips for the initialization, as it’s interactive: don’t enable NAT, use 172.16.0.1 for DNS and “10.0.0.0/16 10.0.0.1/32” for IP range. Although CBSD does great job at figuring out what PF rules it should insert/remove, I like all the rules in one place, /etc/pf.conf:

ext_if = "re0"
jail_if = "lo1"
bridge_if = "bridge1"

set skip on { lo0, $jail_if, $bridge_if }

scrub in all

nat on $ext_if from { ($jail_if:network), ($bridge_if:network) } to any -> ($ext_if)

block in log all
pass out all keep state
pass proto tcp to any port ssh
pass inet proto { icmp, igmp }

You can see lo1 and bridge1 interfaces, so they have to be configured. That’s done in /etc/rc.conf:

cloned_interfaces="bridge1 lo1"
ifconfig_bridge1="description re0 172.16.0.1"
ifconfig_lo1="up"

A bit of an explanation is needed. First, lo1 will be used for jails, bridge1 for VMs. As CBSD does great job of automating VM management, it also creates bridge and tap interface(s). As I wanted the network part to be as static as possible, I’m creating bridge1 the way CBSD would and give it the IP which VMs will use as DNS server and gateway. My only network card on desktop is re0, hence the description. Disclaimer: all this CBSD network mangling is more appropriately done by patching /cbsd/vnet.subr and /cbsd/vnet-tui.subr.

You have to enable PF, of course. I like to do it in /etc/rc.conf.d directory for DevOps purposes I’ll talk about later. There are two files for PF: pf and pflog.

/etc/rc.conf.d/pf:

pf_enable="YES"

/etc/rc.conf.d/pflog:

pflog_enable="YES"

As CBSD doesn’t provide DHCP or DNS for VMs, we’ll use DNSMasq. For devops on your desktop/laptop it’s enough and even for small to mid size cloud setups it would work. On large scale deployments you probably have BIND, already. To enable DNSMasq, first make it start on boot.

/etc/rc.conf.d/dnsmasq:

dnsmasq_enable="YES"

You need to edit the config in /usr/local/etc/dnsmasq.conf. These are the options I changed:

domain=vm
dhcp-range=172.16.0.50,172.16.0.250,12h
interface=lo0
interface=bridge1
bind-interfaces
resolv-file=/usr/local/etc/dnsmasq.d/resolvconf
conf-dir=/usr/local/etc/dnsmasq.d/,*.conf

All options are present in the file initially, but commented out. In short, this config provides DHCP and DNS service only on bridge1 which is dedicated for VMs while allowing for on-the-fly VM info changing by including everything from /usr/local/etc/dnsmaq.d directory which ends with .conf.

For easier URLs, you can use DNSMasq as your DNS server through 127.0.0.1. To do that this is what you need to have in /etc/resolvconf.conf:

name_servers=127.0.0.1
dnsmasq_resolv=/usr/local/etc/dnsmasq.d/resolvconf

Whenever your DNS settings change, resolvconf will write the info into the file which will make DNSMasq re-read it.

The last thing to do is configure BHyve. You’ll need to load the modules and let CBSD take care of the rest, so add these lines to /boot/loader.conf:

zfs_load="YES"
vmm_load="YES"
if_tap_load="YES"
if_bridge_load="YES"
nmdm_load="YES"

ZFS and VMM are self explainatory module names, while TAP and Bridge are two kinds of interfaces used to emulate network stack inside VM. Null Modem or nmdm is used to get the terminal output through a serial line and tmux.

Reboot and you should be able to create new jails and virtual machines:

# cbsd jconstruct-tui
# cbsd jstart <jail>
# cbsd jlogin <jail>
# cbsd jstop <jail>
# cbsd jremove <jail>

With cbsd jconstruct-tui you have to choose lo1 for the interface, once you’re greeted with the dialog based form. Commands for VMs are the same, just prepend them with b instead of j, e.g. cbsd bconstruct-tui. The lo1 interface is only for jails, so for VMs you don’t have to do anything special.

Nice thing about bstart is that it will start bhyve process in tmux so tmux a will open it. Another nice thing is that if booting from CD-ROM image, BHyve will wait for the VNC connection to start the boot process. That means that booting will not start the second you issued cbsd bstart, but once you start vncviewer localhost, so you can see all the messages.

Now, go and play with it a bit. Don’t worry, you don’t need any images or anything, CBSD will download them for you when you select appropriate template.

Lost Passport

Mon, 02 Jan 2017 03:00:00 +0000

My wife and I went from Hamburg to Belgrade on 31st December. We had connected flights and from Hamburg we went to Zurich. In Zurich we realized my wife lost her passport. The situation is hilarious looking back, but totally not fun while it’s happening. So, her passport fell from her pocket in the plane and we noticed that on our way to the customs. What happens next is hell breaking loose, which we kindda expected for several reasons, but let me go back to the begining.

The reason why we were in Hamburg so late in the year was CCC, of course. That means we’re with our friends in Hamburg on the night of 30th, and flying home on the morning of the 31th December. One thing was sure: we won’t get enough sleep. Laying down in 4am and getting up at 6am makes you slugish. We asked a German friend about the S-Bahn to airport, and although it helped, seing everything in German in 7am throws you off the balance. Moreover, S1, which we were waiting, has (at least) 2 platforms which are not next to each other. After some reading we found out which is the right one and which dirrection we need. Now comes the first big problem: first 3 cars don’t end up on the same place as last 3 cars, so in order to get to the airport, you have to enter the right one. Description is, of course, in German. Finding someone awake at 7am on 31th December is hard. Finding someone who speaks German and English at that exact same time is … frustrating, but we made it. Second problem: although the customs work 24/7, the one you need to get you EU tax money back works 9-17!!! There’s no country in the world that doesn’t hide behind bureaucracy when they need to give you (back) money. That was really, really close to our first departure, that I got my money back, got mu bag and entered the plane with ~5 minutes breaks between the actions. Third problem: got in Zurich, no passport. For fuck sake, how big that airport is! It’s even bigger when you have a forth problem: you didn’t go to toilet for 5 days, and your somach decides to do something about it that very minute. Fift problem: wife had an operation last year and running is not what she can do for long. Sixt problem: it’s fucking New Year, no one is working … almost. Luckily, the clerk we saw about lost passport is one very capable woman who managed to find it by having 1000 phone calls. One thing was going on our hand: our plane was comming from Amsterdam where the fog made all flights late around ~40 minutes.

So, after around 40 minutes of running with a feeling of a knife in my stomach, we got into the line for departure. At that moment I’m all swetting like crazy, pushing all my things into my wife’s hands with only 4 words: “I can’t, no longer!”. How do you hurry in that situation? Relax so you mussle doesn’t cramp, or hurry like crazy? I’ll tell you this much: that was the fastest toilet business in my life. OK, everything is fine, I’m in line again, than in plane, we’re in Belgrade … fuck!!! Did that clerk say she called the gate to take our bags off the plane because they can not travel without us? I remember she did that, but did she call back? After ~15 minutes at staring at other people luggage, we saw our bags. Of course they were the last.

Everything is fine now, really. We only need to get home, which has little detour to get a drivers dog so she doesn’t freak out when all the cracks start banging. After that we finally got home! Ah, rest, finally.

Rest my ass! In 4 hours friends are comming, we’re celebrating New Year at our flat. Shower, eat, sleep 2 hours, drink, eat, drink, drink … in 3am pass out. Please, no more exciting and tiresome events untill summer, please!

FreeBSD WiFi and Ethernet Bridging and Aggregation

Sat, 24 Dec 2016 22:27:00 +0000

The title is cumbersome but the topic is quite ordinary life, to be honest. Idea is to have WiFi and Ethernet cards unified. On the router that means that whether you’re getting your address over Wifi or Ethernet, you get the same range. On the client (usually laptop) we want to not be able to distinguish if we’re on Wifi or Ethernet connection, but we do prefer Ethernet.

Router Configuration

When you think about it, what we need is described in lame terms as “I want wifi and ethernet interface to have same IPv4 address”. As that’s kindda imposible, you can have a bridge with Spanning Tree Protocol (or STP for short), which is the next best thing. Bridge is just one virtual interface which binds, or bridges, two or more interfaces. In my case, ethernet interface is re1 and wifi interface is wlan0. To create a bridge with STP (to be totally honest, FreeBSD defaults to RSTP or Rapid STP which is STP compatible) you’ll add the following to /etc/rc.conf:

# WiFi config
wlans_ath0="wlan0"
create_args_wlan0="wlanmode hostap mode 11gn"
ifconfig_wlan0="ssid mywifi channel 8"
hostapd_enable="YES"

# Bridge config
ifconfig_re1="up"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm re1 addm wlan0 stp re1 stp wlan0"
ifconfig_bridge0_alias0="inet 192.168.5.1 netmask 255.255.255.0"

That’s it. You’ll notice that we enabled hostapd, too. It is what provides WPA2 on WiFi. It’s config is:

interface=wlan0
debug=1
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel
ssid=mywifi
wpa=2
wpa_passphrase=password
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP

I found tutorials which set MAC address of all three interfaces: wifi, ethernet and bridge to the same value. I tried it and it doesn’t work. As a matter of fact, the config show here will produce that you see the same MAC of your router no matter how you connect to it: Wifi or Ethernet.

What you probably want is NAT enabled router. My prefered tool is PF. To enable PF and logger, add the following to /etc/rc.conf:

pf_enable="YES"
pflog_enable="YES"
gateway_enable="YES"

The simplest rules for this use case are:

ext_if = "re0"
int_if = "bridge0"

set skip on lo0

scrub in all

nat on $ext_if from $int_if:network to any -> ($ext_if)

block in all
pass out all keep state
pass proto tcp to any port ssh
pass inet proto { icmp, igmp }

Laptop Configuration

What we want on laptop is quite the opposite of the router: instead all bridged interfaces to work at the same time, we want only one to be active. The rule for activation is: if there’s no signal (or carier) on the Ethernet interface, use WiFi. That kind of virtual interface is called lagg or Link Aggregation. One lagg has master interface (in our case Ethernet one) and slaves. Master interface is the one that is added to the lagg the first. To aggregate WiFi and Ethernet on em0 interface, add the following to /etc/rc.conf:

cloned_interfaces="lagg0"
wlans_iwn0="wlan0"
create_args_wlan0="country US regdomain FCC"

ifconfig_lagg0="laggproto failover laggport em0 laggport wlan0 DHCP"
ifconfig_wlan0="ether f0:de:f1:64:2c:3b WPA"
ifconfig_em0="up"

There are 3 important parts. First, wlan0 MAC is set to be the same as the one found on em0. This minifies the time it takes to switch between the two interfaces. Second, don’t put DHCP anywhere except on the lagg0. Third, you have to bring all the interfaces you use up (hence the last line).

With the router that leases the addresses only from one pool, and interfaces that are effectively on the same IP range, aggregating interfaces on laptop with the same MAC address will give you the same IP no matter how you’re connecting. Also, as FreeBSD PF doesn’t have egress, having all outbound traffic on one interface, be it virtual or hardware, makes things easier to filter and route.

FreeBSD Network Optimization and CCC Tickets

Mon, 07 Nov 2016 23:45:00 +0000

Today was the first day of presale tickets for people without voucher. Let me be precise, today at 20:00 started the DDoS on ticket servers. So, let me put things into perspective. Because CCC is more and more popular, they had to do something to limit the number of tickets. They decided to give vouchers to hackerspaces and alikes. As we heard of that too late (or were too lazy, pick your favorite), the reply we got was “we ended requests for vouchers earlier today”. Bummer!

Next you can do is buy a normal ticket, but that’s limited, too. You have 3 chances to do so: 07, 19 and 25 November. So, let’s see. Last year there were about 13k people. Let’s say only 5k are buying tickets. Guess what, they all tried to do it today at exactly 20:00. If you dig/nslookup tickets.events.ccc.de, you’ll see that it’s the only one IP. There might be multiple app servers behind nginx (yeah, we saw nginx 500 error messages), but still, that’s HUGE amount of traffic for one IP. And then games began.

My wife and I loaded the page 30 minutes before, so the CSS, JS and whatnot is cached. We knew there’s going to be problems, so we were prepared. We both have Linux with dual boot: Windows in her case, FreeBSD in mine. I saw it in her eyes “I already booted Windows …” and got reply to silent question “Yeah, and you’re going to race with all the people on better OSes?”. So there we are, Linux on her machine, and FreeBSD on mine. I was thinking two things: she’s already on Linux, if the site doesn’t work on FreeBSD for some reason, there’s her machine to refresh the page while I reboot, and let’s add diversity, who knows maybe Linux has some problems that FreeBSD doesn’t. I was going to go with BSD because they practically implemented TCP/IP sockets (hence the name “BSD sockets”). I know that network stack implementation is better on FreeBSD than Linux, and I don’t even want to discuss Windows. Did it work? Oh yeah! My wife’s Linux didn’t load the first page while I bought us tickets. So let’s see how I did it.

Does network stack and BSD vs. Linux really work? Absolutely NO! The difference is that my wife’s on laptop over WiFi, and I’m on desktop over ethernet. WPA2, I’m sure, is much more overhead than Linux implementation of TCP/IP stack. I’m sorry if you feel fooled by the title, but I really thought about optimal implementation of TCP/IP when I considered which OS to boot, and it’s more fun to have a title like this then “How I bought CCC tickets”.

So, here we are, the page is loaded, I choose to buy 2 tickets (yeah, my wife is a huge fan of hacking) and … error 500. I feel I used all the force in the Universe pressing F5 frantically. Then I got to the page to choose payment method and then one page to enter email, than the final one, which didn’t work. I decided I’m gonna get the tickets no matter what! On the third page, you’re informed the ticket is reserved for you for 30 minutes. If for any reason you don’t want it, you just close the browser and the ticket is returned to the pool of tickets. That’s nasty, as I have to do something with the server which is under heavy DDoS . Of course it was about to timeout when I tried a little hack (it’s not even a hack, to be honest): return to the previous page, click the same button, and after using the F5 force again, you get another 30 minutes. So, I figured how to not be time limited. That’s cool! Then comes the hard part. On the last page where you confirm everything, submit button didn’t work and gave some errors about django in the browser console. WHAT THE FUCKING FUCK!!! Pissed off I looked at #33c3 hashtag on twitter and saw that most people who bought tickets are speaking German. Right! Switch to German on the last page, and F5 force again. IT WORKED!!! Do I speak German? Nein! The button looks the same and I really didn’t care about the letters.

The moral of the story: you better be system and network administrator and backend and frontend developer if you want to buy CCC ticket without the voucher.

Update 1

No matter how you intend to pay for the ticket, select Bank transfer as selecting credit card will give you one more form to fill in. You’ll get the email which, among other details, contains link to tiketing server’s page with details about your ticket. Tomorrow, when all the DDoS is over, go to that link and choose “switch payment method” if you want to pay with credit card. This way you have one page less to display, which is infinity for a server under DDoS.

FreeBSD In Two Months

Fri, 28 Oct 2016 00:15:00 +0000

My last post was about CCC and it’s almost the time for a new one. I was so bussy hacking things that I forgot I have a blog. Huge amount of new things happened to me, but I’ll focus on the FreeBSD. I installed BETA1 of 11.0 as soon as it came out as I really wanted to see how good is the Docker support. Now I don’t even use Docker, and FreeBSD is my main OS on laptop and desktop, although I still have Debian on desktop for the sake of recording my band. So let me give you some context and my history with this operating system.

I remember I was using 5.0 for a while and 4.9 before it for quite a while. The first thing I noticed is the size of the system and the speed at which world is compiled. It was so much faster to compile FreeBSD libc than glibc that I suspected that there’s something missing in the BSD libc. For me, that was all I needed to switch from using SuSE 7.2 at the time (I think). It was 2003-2005 that I used FreeBSD with some pauses. As my university was under heavy influence by M$, I had trouble adjusting Linux to do what was needed. Doing the same on FreeBSD wasn’t even poissible. Soon, I switch to Gentoo, as it was the next best thing.

To be honest, I never looked back to FreeBSD until two months ago. What I read was that version 11 is comming with the Docker support, which I used heavily on my servers. From past experience I knew about the beauty of FreeBSD build system and PF as firewall and ZFS and DTrace and … I was lucky enough that about a month after I installed FreeBSD on my laptop, there was a EuroBSD conference in my country. Of course I rushed there! But something happened since then.

Talking to some of the clients and friends gave me impression that GPLv3 is scaring people and companies. As a huge fan of GNU, I just thought they are not using open source, so they don’t know what they are talking about, but then I read about GPLv3 more. As a tool for binding people to write more open source code, it became monopolistic licence in a way that if anything is GPL, everything is GPL. This resonated in my head for some time and that was what made me realize why are BSD people so licence pure - you can’t ignore business just because you have this vision in your head that says “everyone should write open source code”. BSD community seamed (and it is) more permissive and open to the real world.

But licences are not my strong point, so I’ll stick to the tech part. Let me just briefly describe some of the technologies in the BSD world. First in my book is ZFS. It is so much more than a file system. It has RAID included, if needed, it has copy-on-write, it’s got 128bit system, it has volume manager and on top of it all, RAID is not just RAID, it’s RAIDZ, meaning ZFS keeps extra checksum for every block, which make it super consistent.

Second on my list of favorite technologies is PF. It’s so readable that sometimes I wonder if they screwed up implementation just to make it more readable (of course they didn’t). Even so, it’s strongest point is not syntax, it’s statefulness of the firewall. PF deserves a post on it’s on but to put the statefull firewall into context: it gives you more logical connections between the packets.

Third one is DTrace which stands for “dynamic tracer”. It’s an interesting idea that every OS is full of probes in different places, and when turned on, they give you information about … well, depends what probs you enable. When no probe is enabled, it has no overhead at all, which makes it great for debugging production servers. It can trace kernel and user space and has a AWKish syntax.

Fourth, and last for this post is Jail. Although it’s technology introduced in FreeBSD world in 2000, somehow it didn’t get much publicity. Together with ZFS it makes one hell of a system for hosting stuff. Also, that’s the core of FreeBSD implementation of Docker.

Almost none of that is what I’m working on, right now. Although I made a switch because of the above mentioned technologies, the main reason for me to hack it more is it’s fully preemptive kernel, which is the core of any real time system. Real time makes sound better and delay from plucking a guitar string to hearing it on the speakers is lower. Most of the work on supporting audio interfaces with more than 8 channes is done by Hans Petter Selasky, so I’d like to publicly thank him for all the trouble he when through. Although my FreeBSD DAW is not perfect due to smaller number of audio apps then the number found in Linux world, it shows huge potential.

What happened in 2 months exactly? As I’m Python and JavaScript developer, musician, video editor and in one word hacker, I didn’t expect FreeBSD to fullfill all my needs (no other OS did). Today only Drumgizmo is what’s missing on my FreeBSD box to make it a perfect DAW. To be honest, after two months of using it I expect that I would be learning kernel development in order to make any sound. Instead, it almost gives me all the power I would ever need (yeah, I know there’s new power to be found).

So to put it all together, what FreeBSD gives me is the power to have one machine for everything: security, firewalling, FS consistency, real time audio and development environment for any language I choose to work with. I really enjoy my new OS, but I have one concern. As I’m talking to people how great FreeBSD is, I’m affraid I’ll become one of those preachers who can’t stop talking. It’s not my vocal cords I’m concerned about, as after all I am a singer in a metal band. It’s the damage I might do to the FreeBSD as a project if I’m preaching about it too much.

Thank you, FreeBSD project!

32C3

Wed, 20 Apr 2016 15:00:00 +0000

We got in Hamburg few days before Christmas so we had enough time to see the city, this time. I had a feeling that people in Hamburg are depressed which was a drag as I got there not too happy about the recent development of my life. To be honest, first few days I thought I would be better at home, or anywhere else. But then the congress started and it got a bit better, but the feeling of something missing didn’t go away. Even worse, I wasn’t the only one who had that feeling. As every normal guy would do, I fixed it with rakija, of course.

Although it’s important for me to tell you the whole state of mind I was in, that’s not the reason I’m writing this post. We (LUGoNS) were lucky enough to have a table beside MetaLab guys. Amount of happiness coming from them was amazing. As always, CCC was full of wonderfully talented hackers and I had great time at the Python Booth, for example. Great time passes fast, and the closing ceremony was getting close.

For me, the whole point of going to 32C3 was closing ceremony. It was really emotional for me. Emotional enough to find a shoulder to weep on it. Tears just broke out of me and all bad feelings started to pour out of me. What really happened on that ceremony was that the guy who was talking said something I didn’t know it will be such a trigger = “Look around you, remember small things and take it home”. And I did. That’s when I realized what is going on.

The background story first. Back in 1999. my country was bombed by NATO. Few days of that period I spent in a moist, old basement with my parents, PC motherboard and a magazine with audio synthesizers. That basement had no electricity, toilet or anything, so when something was needed we went to the house. What’s important is that it was full of equipment. At that point I made a promise to myself that I will have huge pile of electronics one day. Meaningful electronics! That “remember the small things” somehow reminded me of that day 16 years ago. After coming to my senses my girlfriend asked me “what the hell was that?”. The only thing I kept repeating was “I made it”. I was repeating it like hypnotized.

I’m a co-founder of a hackerspace and we’re piling wonderful electronics. I barely hold my self together while writing this and remembering = “from the shitty basement to a hackerspace”.

Thank you CCC, I will remember it all!

I am a hacker

Mon, 30 Nov 2015 22:00:00 +0000

What is a hacker? Who the hell knows? On top of that, I’m clamming I’m one of them. What gives me the right? I’m not sure, but in short, I know one thing: hackers are smart and they never give up. Once I heard the smartest man I’ve ever met saying to a developer: “OK, make something, then Meka will do the magic”. That’s the best compliment I’ve ever got. I guess that makes me smart, too. And never giving up … I know I’m that kind of guy. If anyone wants me to do anything I’m remotely interested in, it’s enough I hear “that it’s impossible”. I guess it’s enough proof I’m persistent.

Let’s go from the start. I knew I’m good at math since 3rd grade elementary school. I’ve been the best physics student in elementary, too. I’ve never been the best student, that’s for sure. I was more interested in asking “why” and “how” than having a good grade. I always asked myself “what if we do this differently” and it always gave me bad grades, but somehow, when things go south, I get to answer the tough questions. High school was even worse. I was a bit above the average. University? Totally below average. Of course, if grades mean anything to you. The mean shit to me. And let me take this opportunity to say that except 3 professors, all of them are total crap. One of the proofs is that I’ve got this answer to “Why am I failing programming?”: “Your algorithms are too advanced for professors to get them”. Let me repeat: I was below average.

Today, I’m co-founder of a Tilda Center, Novi Sad’s hackerspace. So, I’m discussing funding and finances with my friend and he told me “Don’t make the same mistake I made with LUGoNS”. What he told me is “don’t invest too much of your own money into it”. How can I not to? I believe in Tilda with my whole heart. So, after a while, I’m telling him something like “Well, yeah, I know what you mean, but I can make it work, trust me. Besides, I know math, physics, electronics, backend and frontend programming, system and network administration, devops, guitar playing, singing, audio and video production and post production, cooking and I’m starting with capoeira trainings this Tuesday. With all this knowledge, I will make it work on Mars, if needed”. I’m not good at all those things equaly, obviously, but I’m refusing to accept I’m not above average at any of those except capoeira.

For the end, let me just tell you I’m not trying to brag around how good I am at anything. This post is to remind me I’m not as stupid as I have tendency to tell people I am, and to remind me of the most important achievement in my life: I AM A HACKER!!! I have to repeat that in order to understand why I’m so different and unaccepted. Thank you for reading!

OpenBSD WiFi

Tue, 18 Aug 2015 22:00:00 +0000

I’m a proud owner of APU. I’m currently running OpenBSD on it. It’s pretty simple config. Everything was “working” until I decided to tighten the screws on the device, as I purchased a new screwdriver. Since then, WiFi is terribly slow. I had ~2s delay between pressing a key and seeing character when logged in over ssh from laptop to desktop. So, I’ve disassembled the device and assembled it all over again, but no luck. Then I realized that I’m using 11b mode. Switching to 11g and priority 0 made it all working well. SSH is more responsive and speed test shows 10Mbit/s instead of ~5Mbit/s it showed previously. I’ve learned my lesson.

Guitar Pickups

Sat, 25 Jul 2015 22:00:00 +0000

When I was searching for my guitar, I read a lot of articles about pickups, too. In Tilda Center we have at least two guys who play guitar beside me, and once we had discussion about this exact topic, which made me believe that there are not enough articles about pickups, types, usages, electronics behind them etc. Or, maybe, the articles are not clear enough. I’ll try to explain.

First, there are few types of different types of pickups. The most famous are single coil and humbucker. Let’s start with single coil.

From this picture you can see that single coil have single line of magnets. If you’ve ever played a guitar with this kind of pickups, you’ve noticed that they have a lot of noise, especially if you have CRT monitor. Single coil have “flat” sound and a bit thin.

Humbuckers, on the other hand, have two lines of magnets. If you’ve ever heard of balanced cable, you know the basic principle: one line of magnets have their polarity N-S, while the other line is S-N. That means that one line of magnets generate sound, while the other line generates inverse sound. In perfect conditions, if you mix those two signals, generated sound will cancel each other out, and the only signal that you’ll end up is the noise generated in magnet coils. As a matter of fact, noise will be doubled, because generated noise is not inversed like the sound. I hope this is clear enough. Now imagine that you invert one of the sounds before mixing. What you’d have is opposite: double the sound, noise canceled out.

The difference in sound is obvious. There’s less noise (or hum) in humbuckers than single coils. As magnets and wires are not perfect, humbucker will cancel out some of the high frequencies. To me, that’s a good thing, as I find single coil sound too piercing.

There are other types of pickups, but they are either single coil or humbucker in it’s essence. For example, EMG and Seymor Duncan have active pickups: the ones which have small integrated amp in them so the signal they output is much higher (think high gain distortion). In order to power up that amp, 9V battery is needed, and is usually placed with the rest of the electronics on the back of the guitar.

To put it into perspective, let me tell you what bands I associate with what types of guitars/pickups. When I think about single coil, it’s either Strat or Tele. Bends that come to my mind playing those guitars are Dire Straights, Eric Clapton. Iron Maiden, for example, uses single coil, but I always felt they are somewhere in the middle: between singles and hums. When I think about humbuckers, Gerry Moore, Clutch and Corrosion of Conformity come to mind, while I think best example of active pickups are Zakk Wylde, Kerry King and Metallica (sorry for all of those being EMG based, I don’t know anyone playing Seymour Duncan’s Blackouts except a friend of mine, and I’m probably putting them in my next guitar)

Tilda Center Summer

Sun, 28 Jun 2015 22:00:00 +0000

As my plans for summer festivals and beaches are ruined, my activity in Tilda Center increased. Pure action-reaction physics at work. :o) We got two new tables and weekly meetings on Mondays and Thursdays. On Mondays is a general discussion or gathering to just hang out. On Thursdays, theme of the meeting is One Love, project a few of us are contributing to. In future, we’re hoping to fund our hackerspace with it. I can tell you it’s interesting. Amount of ideas worth trying out is astonishing. One of the members even brought guitar for slight adjustments. Knowledge is flowing, and it’s great. We even have time for Danube beach, when the weather is kind enough. :o) Soon, I hope we’ll have “conference” room in Tilda ready. We already have a white board on the wall. Ah, I love drawings on the meetings. So much easier to comprehend someone else’s idea if it’s drawn. I tend to teach our young members that everything can be represented as a drawing, and if it can’t, you can make analogy to something from common life. It’s great how much errors are stressed out like this, during planing stage. Reminds me of Brad Victor. We opened back in April and we caused so much joy, but there is so much to explore and so much to learn. What a great ride up to now!

Dockerize Everything

Thu, 18 Jun 2015 22:00:00 +0000

Often I hear the “Dockerize Everything” phrase. To be honest, when ever there’s a new technology, people praise it as “it fixes everything”. I don’t believe that’s the truth about any technology, not even docker. It’s not that I think docker is a bad thing. Hell, I wrote series of blog posts about docker. I just think docker is used where it’s not suited too often. One such example is data container. As it might be great for huge number of servers, it’s not that great for small fleets of servers. For example, if you have nginx and data container such that data container is used as a volume for nginx, every time you change data container because you have new files, nginx has to be restarted. That means you have downtime even if you want to upload just a new css file. That’s not so great. But why would you ignore “normal” directories just because docker exists? Docker can use directory as a volume, so if you upload the same css file to a server’s directory, no downtime is needed. That’s what I did for my blog: have the static files in directory and use nginx docker container.

Just to conclude, it’s important to research new technologies, but it’s equally important to draw the line what should new technology do, and what should old one be used.

pfSense Community

Wed, 17 Jun 2015 22:00:00 +0000

Few days ago I wrote about pfsense vs openbsd and twitted about it. As you can see, it triggered a flamewar, which was never my intention. The reason I continued that flamewar (and one more with the same person on other thread) was the I had nothing better to do, to be honest. I was away from home with my laptop, and I was tired of programming and administration projects. I’ve noticed that OpenBSD community hardly said anything. I don’t understand why anyone would be intimidated with the fact that one person twitted about not using some piece of software. Having that attitude could never attract people. Linux user group I’m member of had few men like that, and we ended up pushing away people who wanted to be part of our group and contribute. Even if people are wrong, they should be let to make mistake, if they decided to try something out.

On the other hand, the arguments presented were funny. For example, one of the arguments was that PF on FreeBSD 11 is faster than the one on OpenBSD 5.6. While this may or may not be true, the fact is that FreeBSD 11 is not released, making it bleeding edge version. At the same time, OpenBSD 5.7 is the current stable version, not 5.6. And who said that the speed of firewall was even important, compared to ease of use and integration with the rest of the system. As a matter of fact, firewall is just one small part of the system, and it’s not even the reason why I made the switch.

All in all, pfSense community, or at least two gentlemen that were the loudest, are one more reason why I will not use their product. It’s not my anger or any other emotion. If I learned anything from open source projects is that community is everything. Sad for pfSense!

pfSense vs OpenBSD

Sun, 07 Jun 2015 22:00:00 +0000

I’ve had pfSense on my router for some time now. To be honest, about a year. The reason I decided to switch to OpenBSD should be obvious, but I’ll write about it anyway. Let me introduce you to the background of the two choices.

PfSense is based on FreeBSD. To be more precise, version 2.1 which was latest stable release up to few months ago, is based on FreeBSD 8, while 2.2 is based on FreeBSD 10. To be honest, FreeBSD is one of the most exciting projects ever. In my opinion mostly because of the technologies it ported from other operating systems like ZFS and DTrace from OpenSolaris and PF from OpenBSD, to name the few. With such diversity of ported projects, it has my deep respect if for nothing else, than for being able to incorporate and maintain them in a secure and stable way. If you ever tried to continuously port and maintain a software from other platform, you know what it takes. For me, firewall is the most important part of routing OS, and FreeBSD having incorporated the best one, makes it great.

But, it’s not all that great. For example, PF in FreeBSD is based on OpenBSD’s implementation (which is original) from 4 years ago. Although FreeBSD and OpenBSD have different plans and courses for their implementation of PF, I like OpenBSD’s syntax better. Another thing that made me switch to OpenBSD is DHCP+DNS on pfSense. I don’t know if it’s up to FreeBSD or pfSense, but once a machine gets IP, it takes too long (as in 15 minutes) for it to be registered in DNS. All of this was enough for me to at least try OpenBSD. Hmmmm … try. Let me tell you I’m not going back to pfSense, but I might give FreeBSD a chance. I like to experiment, so we’ll see.

I’ve also ditched ubound and dhcpd in favor of DNSMasq which serves great as a DHCP+DNS server. WiFi hostap works like a charm, and almost every system config file is in /etc. The way to make console redirect to serial port is to enter 2 lines in /etc/boot.conf (you have to use proper pfSense image to do that). All that and the fact that Cisco and Apple is PF makes me feel warm. :o) No, seriously, what makes me have this feeling of security is what I’ve heard from a friend few years back: “OpenBSD policy that the mistake in the documentation is the mistake in the code”.

One of these days I’ll be writing a script to automate the provisioning of my router which will portrait the ease of using OpenBSD more than this post.

Tube vs Transistor

Sun, 07 Jun 2015 22:00:00 +0000

For so long I’ve listened about debate how tubes have “warmer” sound. All audiophiles will tell you that you can not compare amps which are tube based to those that have transistors in them. I don’t say they’re wrong, but I’m never satisfied with “It’s just the way it is” explanation, so I searched for more info. First, I don’t believe that just because you use tubes you get better sound. There has to be more to it. As they’ve explained me, tubes have a feature which makes even harmonics stand out, while the transistor does the same for odd ones. This makes tube harmonics get along better. For simplicity, think of overtones which are octaves. Octaves are 2x, 4x, 8x … the original sound which are even harmonics.

Now the fun part. I didn’t know, but more than once I’ve heard from really educated people that you can make an amp with transistors which makes even harmonics stand out. You’ll probably ask yourself, just like I did, why don’t we have these amps around. I don’t know, but I’m almost sure it has something to do with economy.

There is some area of audio amps where tubes are superb and transistors can’t ever come close to them: distortion in guitar amps. The reason for this is that tubes and transistors act totally different when they are saturated, hence the difference in sound of distortion.

I’m a guitar player, among other things, and the best think I can think of, in theory, is tube preamp with transistor power amp which makes even harmonics stand out. I don’t know if it’s possible, but I would sure would like to try to create one.

Sat, 06 Jun 2015 22:00:00 +0000

PDF

Goran Mekić

Senior Backend Systems Engineer | Low-Latency & Distributed Systems | Rust, C++, Python

Personal Statement

Senior backend engineer with 15+ years of experience designing, optimizing, and operating high-performance distributed systems in production environments. Specializes in low-latency processing, system-level optimization, and debugging complex interactions across compute, networking, and storage layers.

Proven track record of improving performance and reliability in large-scale infrastructure, including systems managing 250+ PB of data. Strong background in real-time processing (audio/DSP), with deep understanding of latency, throughput, and resource efficiency constraints.

Operates comfortably across the full system stack, from kernel-level behavior and storage systems to backend services and distributed orchestration.

C++ lectures in Serbian
DaFED #61 – Kako automatizacija procesa pomaže u razvoju sigurnijih aplikacija → Presented how process automation improves application security and reduces human error in development workflows
FOSDEM 2023 – Reggae: cool way of managing jails/VMs on FreeBSD → Introduced Reggae as a tool for managing FreeBSD jails and virtual machines, focusing on scalable development environments
EuroBSD 2022 – FreeBSD audio in amateur and professional setups → Explored FreeBSD-based audio workflows across hobbyist and professional use cases, including system configuration and performance
FOSDEM 2021 – Managing virtual resources with CBSD, and beyond → Demonstrated advanced resource management and orchestration using CBSD for virtualization on FreeBSD
BSides Ljubljana 2019 – How Did We Teach (Ourselves) Security → Shared practical approaches to self-learning and teaching cybersecurity within community-driven environments
FOSDEM 2019 – FreeBSD in Audio Studio → Presented real-world usage of FreeBSD in audio production environments, covering stability and performance considerations
BSides Ljubljana 2018 – Modern and Secure DevOps on FreeBSD → Covered secure DevOps practices using FreeBSD, including isolation, automation, and system hardening
FOSDEM 2018 – CBSD Isolation Manager → Introduced CBSD as a solution for lightweight virtualization and system isolation on FreeBSD
BalCCon 2016 – FLOSS Way of Funding a Hackerspace → Discussed sustainable funding models for open-source communities and hackerspaces
BalCCon 2015 – What is Hackerspace → Introduced the concept, structure, and community impact of hackerspaces

High-Impact Contributions

Led performance optimization efforts across distributed systems handling 250+ PB of data, improving reliability and reducing system bottlenecks
Diagnosed and resolved complex system issues including deadlocks, race conditions, and cross-layer performance degradation
Built and optimized low-latency processing systems in Rust and C++ for real-time environments
Designed backend and infrastructure systems that scale across hundreds of nodes with high availability
Improved observability and debugging capabilities using advanced tracing and monitoring tools
Consistently trusted with production-critical systems where failure has significant operational impact

Technical Scope

Languages: Rust, C++, Python, TypeScript
Systems: FreeBSD, Linux, ZFS
Distributed Systems: AWS, Docker, Ansible, SaltStack
Performance & Observability: DTrace, Prometheus, Grafana
Backend Systems: FastAPI, Flask, REST APIs
Specialized Domains: Real-time processing, DSP systems, virtualization

AI & Systems Thinking

Strong understanding of constraints relevant to modern AI systems including latency, throughput, and resource usage
Experience working with data-intensive and real-time processing systems applicable to speech and inference workloads
Regularly integrates AI tools into engineering workflows for faster iteration and problem solving

Engineering Approach

Focus on building systems that are predictable, debuggable, and performant under real-world conditions
Preference for simple, reliable solutions over unnecessary complexity
Comfortable operating in fast-moving environments with evolving requirements
Strong bias toward understanding system behavior at a deep level

Work Experience

SysIT | 2026

Business Development & Technical Consultant

Advise organizations on aligning technology strategy with business objectives, with a focus on scalable and efficient systems
Design and implement AI-driven solutions using tools such as Claude, ChatGPT, and Gemini to automate workflows and improve decision-making
Architect, deploy, and manage containerized infrastructure using Kubernetes, ensuring reliability, scalability, and cost efficiency
Develop and maintain infrastructure as code using Terraform to enable repeatable, secure, and automated environments
Deliver high-performance systems development in Rust, including low-level and DSP-focused applications
Perform kernel-level debugging and patching in FreeBSD environments to resolve complex system issues
Analyze and optimize system and application performance using advanced tools such as DTrace
Support engineering teams in improving development processes, reducing technical debt, and accelerating delivery cycles
Partner with stakeholders to identify inefficiencies and implement technical solutions that streamline operations and improve product performance

Bakson LTD | 2025

Led the modernization of OSS deployment pipelines and infrastructure management through Terraform, Kubernetes, and proactive technical guidance.

Contributed to variety of OSS projects in different programming languages
Worked with a variety of teams to maintain high standards of quality across OSS projects
Collaborated with software developers, system administrators, and IT staff to streamline development and deployment workflows
Set up and maintenance cloud infrastructure required to run specific OSS projects
Designed, implemented, and managed CI/CD pipelines for efficient software delivery
Mentored junior team members and provide technical guidance
Administered kubernetes cluster
Maintained terraform code

Axcient | 2022 – 2025

Owned performance, reliability, and scalability of large-scale distributed storage systems operating across multiple data centers.

Drove system-level performance improvements across infrastructure managing 200+ PB of data
Investigated and resolved critical production issues involving distributed coordination, storage systems, and compute interactions
Performed deep analysis of system behavior using tracing tools (DTrace) and monitoring systems
Built automation for managing infrastructure across hundreds of machines using SaltStack
Improved observability using Prometheus and Grafana, enabling faster diagnosis of production issues
Partnered with engineering teams to improve system architecture and operational reliability
Systems Architect / Backend Engineer

Fourity | 2021 – 2022

Led design and implementation of backend and distributed systems across multiple projects.

Architected scalable backend systems with focus on performance, resilience, and maintainability
Designed distributed system components and infrastructure automation strategies
Delivered systems capable of handling high-load environments with minimal operational overhead
Provided technical leadership and guided engineering decisions across teams
Senior Backend Engineer

Firebird Networks Pulsar | 2018 - 2019

As the sole architect and developer for the Firebird Networks Pulsar project, I was responsible for the design and development of a network gear configuration automation tool. Using Ansible, I automated the setup and management of network infrastructure, making future deployments easier and faster for the company. My role included the full software development lifecycle, from requirements gathering to implementation and testing. Additionally, I worked on the frontend development with React and integrated real-time features using WebSockets for live monitoring and control of network gear.

Key Responsibilities:

Architecture Design: Designed and implemented a scalable solution for automating network gear configurations, improving deployment speed and reducing human error.
Full Stack Development: Developed both frontend and backend components of the system, using React and Flask to create a user-friendly interface and a robust API for network management.
Real-Time Monitoring: Integrated real-time monitoring capabilities, allowing users to monitor and control their network gear from the web interface using WebSockets and Socket.IO.
Deployment Automation: Leveraged Ansible to automate the deployment and configuration of network gear, reducing the time spent on manual setup and ensuring consistency across environments.

Technologies Used: React, Flask, Ansible, Python, WebSockets, Socket.IO

Cala Health | 2018

Worked on a regulated medical platform requiring high reliability, performance, and secure data handling.

Designed and developed backend services supporting real-time, data-intensive medical workflows
Built and maintained REST APIs used across internal systems and external integrations
Improved system performance through profiling and optimization of backend services
Contributed to architecture decisions around scalability, fault tolerance, and system reliability
Implemented secure data handling aligned with medical and privacy requirements
Collaborated with product, research, and clinical teams to deliver reliable system behavior
Debugged production issues with focus on stability and predictable performance under load
Integrated AWS services (S3, Cognito) for secure storage, authentication, and scalability

GeoSolutions | 2017

During my time at GeoSolutions, I contributed to the development and implementation of a custom monitoring solution for GeoNode, an open-source platform for geospatial data management. My work focused on optimizing the performance of GeoNode by integrating a robust monitoring system with Django/Python, React, and D3.js for data visualization. This solution helped provide actionable insights into system performance and user engagement, ultimately improving the user experience for clients.

Key Responsibilities:

Monitoring Solution Development: Designed and implemented a custom monitoring system that tracked system performance and user interactions with GeoNode, using Django/Python and React for the web application layer.
Data Visualization: Leveraged D3.js to create interactive and meaningful visualizations that helped clients better understand the state of their geospatial data and system performance.

Technologies Used: Django/Python, React, D3.js, GeoNode

Photoslurp | 2015 - 2016

At Photoslurp, I was part of a dynamic team responsible for development and maintainance of a platform designed for the collection, exchange, and management of media rights for photos and artwork. As a full-stack developer, I played a pivotal role in both frontend and backend development. My main focus was ensuring that the platform could efficiently handle media rights requests and facilitate the acquisition of media through a seamless user interface. I worked closely with designers to implement interactive elements and with backend engineers to ensure data integrity, efficient workflows, and real-time processing of media requests.

Key Responsibilities:

Frontend Development: Participated in the development of user-friendly frontend features using Bootstrap and jQuery to ensure the platform was visually appealing and easy to navigate. I also integrated Underscore and Backbone.js to facilitate more dynamic user interactions, allowing users to manage media rights seamlessly.
Backend Development: Contributed to the backend development of the platform using Django/Python, which was crucial for managing media data, user permissions, and rights. I also worked on the integration of Celery for task scheduling and asynchronous processing of media requests.
API Development: Designed and built RESTful APIs to handle media rights requests, ensuring that they were easy to use, secure, and scalable. This included building endpoints for uploading, retrieving, and managing media data.
Continuous Deployment: Worked with Ansible to automate the deployment of the platform, enabling continuous integration and delivery (CI/CD) to enhance the efficiency of updates and maintenance.
Backend Optimization: Implemented performance optimizations on the backend by leveraging Redis and RabbitMQ for task queueing and message brokering, ensuring fast and reliable media rights transactions.

Technologies Used: Django/Python, Bootstrap, jQuery, Backbone.js, Underscore, Celery, REST API, Ansible, Redis, RabbitMQ, PostgreSQL

fSD | 2015

At fSD, I was tasked with implementing and automating DevOps practices for the development team working on an ERP system based on PHP and EmberJS. The project required a focus on streamlining the development, deployment, and operations processes to ensure a consistent, secure, and scalable environment. I worked closely with the development and infrastructure teams to integrate modern DevOps tools and security best practices, ultimately enhancing the reliability and security of the system.

Key Responsibilities:

DevOps Automation: Implemented Ansible to automate various development workflows, including server provisioning, configuration management, and deployment pipelines, allowing the development team to focus more on coding and less on manual processes.
Infrastructure Management: Designed and deployed infrastructure solutions to ensure efficient and stable environments for development, staging, and production. I used Vagrant to manage virtualized development environments, improving consistency across the team and reducing the “works on my machine” issues.
CI/CD Pipeline: Integrated Jenkins for continuous integration and delivery, ensuring that code changes were automatically tested, built, and deployed. This reduced deployment errors and enabled faster release cycles.

Technologies Used: Ansible, Vagrant, PHP, Percona, Nginx, Jenkins, EmberJS

Reciprocity Labs | 2014 - 2015

At Reciprocity Labs, I was brought in as a backend developer with a focus on implementing secure authentication mechanisms, particularly SAML (Security Assertion Markup Language) authentication. The primary goal was to integrate a robust identity management system for our clients to enable secure and seamless single sign-on (SSO) functionality across multiple applications. Over time, my responsibilities expanded to include frontend work, and I contributed to the full-stack development of the project. My role involved improving security practices, enhancing application performance, and collaborating with teams to ensure secure, scalable, and efficient architecture.

Key Responsibilities:

SAML Authentication Implementation: Led the implementation of SAML authentication for the project, which allowed the integration of a secure and scalable single sign-on (SSO) solution. This involved working with various identity providers and ensuring the correct configuration of authentication flows.
Backend Development: Developed the core backend services using Python and Flask, building secure RESTful APIs for handling user data, authentication, and application logic. I worked closely with the team to ensure that the APIs were well-documented, secure, and easy to integrate.
Security Best Practices: Served as a security advisor, helping to implement server hardening measures to protect against common vulnerabilities. This included setting up proper authentication, encryption protocols, and access control lists (ACLs), among other security measures.
Infrastructure & Automation: Utilized Vagrant and Ansible to automate development and deployment processes, ensuring consistency and ease of provisioning across development, staging, and production environments. This allowed the team to focus on development while maintaining a high level of infrastructure automation.
Frontend Development: As the project evolved, I took on frontend responsibilities, working with the team to improve the user interface using JavaScript, CSS, and other frontend technologies. I worked on user experience improvements, ensuring the interface was intuitive and secure for users.
Database Integration: Assisted with the integration of MySQL for data storage and worked on optimizing database queries and schema design to ensure the platform could scale as the user base grew.

Technologies Used: Python (Flask), SAML, RESTful APIs, Vagrant, Ansible, MySQL, JavaScript, CSS, Server Hardening, Git

Lambda Solutions | 2014 - 2014

At Lambda Solutions, I served as the primary System Administrator responsible for the deployment, configuration, and maintenance of hosting solutions for Moodle and Totara, two leading open-source learning management systems (LMS). The role required a strong focus on managing scalable, high-performance infrastructure that could support educational institutions and organizations. My responsibilities spanned across system provisioning, automation, and ensuring the systems remained secure, stable, and optimized to handle the traffic and data storage demands of e-learning environments.

Key Responsibilities:

System Deployment & Configuration: Deployed and configured highly available and secure environments for Moodle and Totara, ensuring that these LMS platforms could scale and handle large volumes of users, content, and activity. I implemented best practices in system administration to ensure both systems were running optimally and securely.
Infrastructure Automation: Utilized Ansible to automate the provisioning and configuration of servers. This reduced the time required to deploy and configure new environments, increasing the efficiency of the team and improving the consistency of the infrastructure.
Cloud Management: Worked with Amazon Web Services (AWS) to deploy scalable cloud-based solutions. I set up and managed AWS resources such as EC2 instances, S3 buckets for storage, and RDS for databases, ensuring that the infrastructure was cost-effective and scalable based on the needs of the clients.
System Monitoring & Troubleshooting: Implemented monitoring tools and processes to ensure system health and uptime. I worked with tools such as Nagios and Zabbix to continuously monitor server performance, alerting the team to potential issues before they impacted end-users. I also handled troubleshooting of any system failures or performance degradation.
Security Management: Applied security patches, set up firewalls, and configured SSL/TLS encryption to protect sensitive data. I ensured that best practices in system security were followed, including hardening the servers and ensuring that only authorized access was granted to the systems.
Backup & Disaster Recovery: Designed and implemented backup strategies to ensure data integrity and availability. I worked with AWS S3 for remote backups and configured automated daily backups to minimize the risk of data loss in case of failure.
Collaboration & Documentation: Worked closely with the development team to troubleshoot and resolve any issues related to the integration of Moodle and Totara into the client’s environment. I also created and maintained comprehensive documentation to support ongoing operations and future deployments.

Technologies Used: Ansible, Moodle, Totara, AWS (EC2, S3, RDS), Linux (Debian/CentOS), SSL/TLS, Nagios, Zabbix, Vagrant, MySQL

Secent LLC | 2013 - 2014

At Secent LLC, I worked as a Team Lead and Senior Developer for an advanced reverse proxy and single sign-on (SSO) solution designed for a major electrical company. The project focused on building a highly configurable reverse proxy system that could integrate seamlessly with the client’s infrastructure to manage authentication, access control, and secure communications across multiple internal and external systems. Additionally, I contributed to several critical tasks involving development, security, and automation to ensure the solution met the company’s rigorous security and scalability requirements.

Key Responsibilities:

Project Leadership: As the team lead, I managed a group of developers and collaborated closely with system administrators to design and implement a flexible, scalable reverse proxy system. I was responsible for coordinating the development efforts and ensuring that the team adhered to deadlines, followed best practices, and delivered high-quality code.
Single Sign-On (SSO) Solution: Led the integration of a Single Sign-On (SSO) solution based on Django/Python, which allowed the company to authenticate users across various internal and external systems using a single set of credentials. This implementation was built to be highly secure and scalable, ensuring that sensitive data was protected at all times.
Backend Development: As a senior developer, I worked extensively on the backend of the solution, leveraging Django/Python to build custom RESTful APIs and integrate SimpleSAMLphp for authentication. I was also responsible for optimizing the performance of the backend and ensuring it could handle the high volume of authentication requests securely.
Security Enhancements: Contributed as a security advisor, ensuring that all security best practices were followed throughout the development process. I helped implement secure data transmission using SSL/TLS, enforced access control policies, and conducted regular security audits to identify and mitigate potential vulnerabilities.
Automating Deployment: Utilized Ansible to automate the configuration and deployment of servers. This allowed the team to quickly spin up and configure new environments, ensuring that each deployment was consistent and secure. I also worked with Git for version control and collaborated with the team to maintain an efficient development workflow.
Cross-Functional Collaboration: Worked closely with other teams, including the DevOps and network security teams, to ensure that the reverse proxy system integrated smoothly with other infrastructure components. I also provided support for troubleshooting and resolving integration issues across the organization.
Database Integration & Management: Managed the configuration of databases such as PostgreSQL to store user authentication data, ensuring that it was encrypted and compliant with company security policies. I helped ensure that the database was optimized for performance and could scale with the growing user base.

Technologies Used: Django/Python, REST API, SimpleSAMLphp, Git, SSL/TLS, Ansible, PostgreSQL, Vagrant, Apache, AWS

I worked on a project for an aerospace company that involved testing software and building packages for in-house applications. This role focused heavily on ensuring the quality, performance, and reliability of the software systems while also contributing to packaging and deployment processes. My responsibilities included creating robust testing strategies, managing software builds, and automating deployment pipelines to ensure that software releases were stable and met all specified requirements.

Key Responsibilities:

Software Testing & QA: Worked closely with the development team to test and validate software for the aerospace company’s in-house projects. I designed and executed various types of tests, including functional, performance, and regression tests, to ensure the software met the required standards and specifications.
Build Automation: Assisted in the creation and maintenance of automated build systems for software deployment. I utilized tools such as Jenkins to streamline the build and testing processes, reducing the time required to validate new software releases and ensuring that any issues could be quickly identified.
Package Management: Created packages for software deployment on various platforms, ensuring that the installations were seamless and reliable. This involved working with VMWare and other virtualization technologies to test the software in isolated environments before deployment to production.
Version Control & Collaboration: Collaborated with developers to manage software versions and handle source code integration using Git. I helped manage code repositories and maintained proper versioning to ensure that software releases were traceable and reproducible.
Deployment & Release Management: Contributed to the planning and execution of software deployments, ensuring that the release processes were smooth and that any necessary configuration changes were properly documented. I also helped prepare documentation related to the deployment and configuration of software packages.
Issue Resolution & Troubleshooting: Worked with the development team to troubleshoot issues identified during testing. I helped debug and resolve software-related issues, ensuring that fixes were applied and retested promptly before final release.

Technologies Used: Jenkins, VMWare, Git, Redmine, Software Testing (manual and automated), Build Automation

Saturized | 2011 - 2013

At Saturized, I was responsible for ensuring the reliability, performance, and security of the company’s infrastructure. My role focused on capacity planning, storage allocation, system backups, security management, and troubleshooting recurring issues. The systems I managed were crucial to the company’s operations, and I played a key role in ensuring that all technical requirements were met while planning for future scalability and redundancy.

Key Responsibilities:

Capacity Planning & Storage Management: Managed and planned for system growth by allocating storage and ensuring the infrastructure could scale to meet the company’s needs. I handled the storage requirements for databases and file servers, ensuring that the infrastructure could accommodate increasing data loads while maintaining performance and redundancy.
System Monitoring & Performance Tuning: Implemented monitoring tools such as Munin, Icinga, and Zabbix to proactively track system health and performance. I was responsible for identifying performance bottlenecks, optimizing system configurations, and resolving issues before they impacted operations. This included fine-tuning server configurations for optimal performance and reliability.
Backup & Disaster Recovery: Designed and implemented backup strategies to ensure the integrity and safety of critical data. I ensured that backup solutions were in place for both file and database storage, and I set up disaster recovery plans to minimize downtime in the event of a failure.
Security Management & Compliance: Worked closely with the security team to implement strong security measures, including setting up firewalls, applying security patches, and ensuring proper access controls. I was also responsible for maintaining system hardening protocols to ensure compliance with security standards and protect against unauthorized access.
Server Maintenance & Administration: Managed various servers, including web servers, file servers, and firewalls. I was responsible for the ongoing maintenance, configuration, and troubleshooting of these servers, ensuring that they remained secure, reliable, and available for company operations. I used Puppet for configuration management to ensure that servers were consistently configured across the infrastructure.
Virtualization Management: Worked with QEmu/Libvirt to set up and manage virtual machines (VMs), ensuring that resources were efficiently utilized and that virtual environments were secure and properly isolated. I also assisted with configuring and managing Gerrit and Git for source code version control.
Team Collaboration & Root-Cause Analysis: Worked closely with the development and network teams to conduct root-cause analysis of recurring system issues. I identified patterns of failures, developed solutions to address them, and collaborated with other teams to implement long-term fixes.

Technologies Used: PostgreSQL (Master/Slave Replication), QEmu/Libvirt, Gerrit, Git, Redmine, Munin, Icinga, Zabbix, Puppet, Firewall Management, Jenkins

DDOR | 2006 - 2011

At DDOR, I was responsible for the administration and management of Solaris and Linux systems, focusing on configuring, troubleshooting, and maintaining a variety of services and applications. I played an essential role in ensuring that the company’s IT infrastructure was secure, reliable, and scalable. My work also included system backup and recovery, capacity planning, performance tuning, and providing support for a wide range of internal systems.

Key Responsibilities:

Solaris & Linux System Administration: Managed both Solaris and Linux servers for a wide range of business-critical applications. This included the installation, configuration, and maintenance of various services, as well as ensuring optimal performance and security across the entire infrastructure.
Storage Management: Administered enterprise storage solutions, including EMC Clariion and Fujitsu Eternus DX440 storage systems, ensuring data redundancy, high availability, and performance. I was responsible for managing storage volumes, setting up LUNs (Logical Unit Numbers), and ensuring that the storage infrastructure was optimized for both performance and reliability.
Virtualization & Cloud Infrastructure: Managed virtualization environments using VMware ESXi, where I configured and maintained virtual machines (VMs) and their corresponding resources. This included setting up virtual networks and ensuring that VMs were properly configured for maximum resource utilization and system security.
Backup & Disaster Recovery: Responsible for configuring and managing backup and recovery processes to protect critical company data. I implemented regular backup routines, tested recovery procedures, and ensured that backup data was always available and accessible in case of system failure.
Performance Tuning & Capacity Planning: Led efforts for performance tuning, identifying system bottlenecks, and optimizing server configurations to ensure optimal system performance. I also performed capacity planning, ensuring that the company’s infrastructure could scale with the growing demands of the business.
Security & Monitoring: Worked closely with the security team to implement robust security measures, including configuring firewalls, ensuring the proper application of security patches, and auditing system logs for unauthorized access. I used monitoring tools such as Monit and Munin to track system health and alert the team to potential issues before they became critical.
Troubleshooting & Support: Provided ongoing system support and troubleshooting services for various IT systems, resolving issues related to hardware, software, and network configurations. I worked with other teams to ensure that issues were resolved quickly, minimizing downtime and disruption to business operations.
Configuration Management & Automation: Used Puppet for managing the configuration of servers and ensuring that all systems were consistent, properly configured, and compliant with internal policies. I also contributed to the automation of various system management tasks to reduce manual work and improve overall efficiency.
Application Support: Supported applications such as Oracle APP and Tomcat by configuring, monitoring, and troubleshooting performance issues to ensure seamless operation. I ensured that the applications were secure and accessible, with minimal downtime.

Technologies Used: Solaris, Linux, EMC Clariion, Fujitsu Eternus DX440, VMware ESXi, Monit, Munin, Puppet, Oracle APP, Tomcat, Subversion

Home Invest | 2006 - 2006

At Home Invest, I was responsible for the implementation, administration, and ongoing maintenance of various network services, including Mail, DHCP, DNS, and Domain Controller. This role required ensuring that the infrastructure was secure, efficient, and well-maintained, with a focus on Linux-based solutions and firewall security. I contributed to optimizing and automating the company’s IT operations, ensuring smooth communication and network services across the organization.

Key Responsibilities:

Network Services Management: Installed, configured, and maintained critical network services including Mail (Postfix, Dovecot), DHCP, DNS (Bind), and Domain Controllers. This ensured that all network communications and services were running efficiently, securely, and with minimal downtime.
Linux-based System Administration: Led the administration of Linux systems that served as the foundation for the company’s network infrastructure. Ensured proper configuration, updates, and security patches were applied in a timely manner to keep systems secure and functional.
Firewall Configuration & Security: Managed network security using IPTables to set up firewalls, securing the company’s internal and external communications. I configured firewall rules to prevent unauthorized access, protect against external threats, and enforce strict access control policies.
Email System Setup & Maintenance: Configured and maintained the company’s email servers, ensuring reliable communication both internally and externally. I configured Postfix for mail sending, Dovecot for IMAP, and applied necessary filters to ensure security, such as anti-spam measures.
DNS Setup & Maintenance: Managed the company’s DNS servers using Bind, ensuring that the domain names were properly configured and that DNS queries were resolved quickly and accurately. This was crucial to maintaining seamless internet connectivity and access to company resources.
User Account & Permission Management: Configured user accounts and permissions, ensuring that employees had the appropriate access to the network resources they needed while maintaining a secure environment. This included managing access to network shares, email services, and domain resources.
System Monitoring & Troubleshooting: Monitored system health and performance, ensuring that all services were up and running. Addressed any issues related to system performance or network downtime and applied solutions in a timely manner to minimize disruptions.
Documentation & Knowledge Transfer: Created comprehensive documentation outlining system configurations, network setups, and maintenance processes to ensure that procedures were clear and easily replicable in the event of system changes or staff turnover.

Technologies Used: Linux, Postfix, Dovecot, DHCP, DNS (Bind), Samba, IPTables

Projects

Maolan – Real-Time Audio Processing Platform (Rust) | 2018

Designed and built a high-performance audio processing system focused on low-latency workloads.

Architected a modular audio processing engine in Rust with emphasis on performance and extensibility
Implemented real-time processing pipelines with strict latency constraints
Optimized system behavior across Linux and FreeBSD environments
Integrated multiple audio backends and drivers for stable and efficient processing
Worked at OS and hardware interaction level to minimize processing overhead
Distributed Communications Infrastructure

Tilda Center | 2015

Tilda Center is a hackerspace and educational hub focused on modern technologies and open-source tools. I co-founded the center to bridge the gap between affordable, high-quality education and cutting-edge technology. I am responsible for leading technical research, teaching, and system administration.

Developed and delivered training in Python/Flask/FastAPI, React/Redux/Svelte, HTML/CSS, electronics, and audio production.
Managed IT infrastructure, including FreeBSD/Linux systems, ZFS, PF, and automation using Ansible.
Led research initiatives on FreeBSD kernel development, DSP/RT Rust/C++ environments, and audio/music systems.
Designed and implemented internal systems for virtualized environments and efficient resource management using CBSD/Reggae.

Communications Server | 2018

Designed and automated deployment of production-grade communication systems.

Built distributed systems for mail, DNS, chat, and directory services
Designed high-availability configurations with redundancy and failover mechanisms
Automated infrastructure using Ansible and FreeBSD jails
Improved reliability and operational efficiency through automation and system design

CBSD/Reggae/ClonOS - Virtualization & Orchestration | 2017

Worked on tooling for managing large-scale virtualized environments.

Developed orchestration tools for multi-node infrastructure
Implemented parallel build and deployment systems for jails and virtual machines
Optimized resource utilization and system performance in multi-tenant environments
Improved scalability and operational efficiency of virtualization workflows
Storage & System Optimization Work

Drumgizmo | 2017

DrumGizmo is an open-source, multichannel drum plugin and stand-alone application for music production.

Contributed to the FreeBSD OSS audio driver backend and ported the software to FreeBSD.
Maintained FreeBSD ports for seamless integration with the system.

One Love | 2014

One Love is a decentralized management platform for data centers using Docker and cloud technologies. As a full-stack developer, I’ve contributed to both frontend and backend development.

Designed and developed the frontend using React, Redux, and Webpack to ensure seamless user experience.
Developed backend services using Flask, ZeroMQ, and REST APIs for high-performance, scalable communication.
Managed deployment pipelines on AWS, DigitalOcean, and other cloud platforms, with a focus on automation and efficiency.

Education

Master of Science in Business Information Technology

Faculty of Sciences, University Of Novi Sad

Studying all aspect of IT including Computer Networks, Software Engineering, Data Structures and Algorithms, E-business and Artificial Intelligence

Electronics Technician

The electricity & electronics middle school “Mihajlo Pupin” Novi Sad

Mihajlo Pupin Novi Sad Eletronics Middle School Designing and building of electrical circuits, including simple digital boards and fixing all non-SMP low voltage devices, scheme and PCB design, primarily for audio devices.

All Strings Attached

Sat, 06 Jun 2015 22:00:00 +0000

First of all, I’m no physicist. Everything written here is what I’ve gathered from other sources, like books and Wikipedia. As I never had a chance to check my thoughts with someone with proper and formal physics education, I might be (terribly) wrong, so don’t quote me as a scientist or a proven theorist.

Most of my thoughts come from Brian Greene’s Fabric of the Cosmos. I like how the book is easy to understand so much that I recommend it to everyone. I’ll admit it, I was hardly waiting to get to the strings. Don’t get me wrong, the whole book is golden, but I was mostly interested in strings. Just imagine (if you can, but up to now, nobody was able to) that vibrating string manifests itself as a particle. I mean, different vibrational patterns make up different things in our Universe: particles, time, energy, radiation, … Once more, strings create time. Wow! What a concept. But I want to talk about something more scientific, if I may call it that. Throughout the research which is mostly mathematical, scientists came to conclude that the string theory gave birth to 5 different frameworks of string theory. It’s like you say there are 5 different variations of the same theory. A bit awkward situation for science, but the 6th variation solved it all, or at least is trying to do that. It’s not the 6th variation, it’s “one ring to rule them all”, to be a bit more precise. It’s called the M-theory and nobody knows what M stands for. The craziest proposal I’ve heard of what M is: 180 degrees of W which stands for Witten, the person who came up with the idea of M-theory. Edward Witten is such a crazy guy that you have to love him. Although he’s physicist, he’s the first man who got the Fields Medal award, which is award in math. Unbelievable guy!

Anyway, back to the topic. In short, there are strings which are loops, and strings which are like a rope with two ends. Researchers found that “rope like” strings are attached to 2 dimensional branes. I won’t go into what branes are for two reasons: I don’t know and it’s not that important at this time. The funny thing is that all forces are rope alike, except gravity. Even more, all forces except gravity are strings with two ends on the same brane. Now, this is astonishing, at least to me. This makes gravity special, which researchers knew before string theory, but it’s even more special now. In theory, strings are 1-dimensional objects living in a 11-dimensional environment (10 spacial and 1 time dimension). All vibrational patterns of all known forces and matter are bound to the same brane, except gravity. This makes gravity able to traverse our 3 spacial dimensions and span on other branes. How? Gravity is made of loop strings, hence not being tied to any brane. Isn’t that remarkable? I’m thrilled that we can (at least in theory) communicate with other dimensions through gravity. How? First thought is gravitational wave. Imagine that you can use gravitational waves to create bits, just like computers use electromagnetic field. I don’t even know if it’s theoretically possible, but dreaming about other dimensions is becoming less of a dream and more of a research. That makes me happy!

Goran Mekić

Sat, 06 Jun 2015 22:00:00 +0000

Co-founder of Tilda Center - Hackerspace in Novi Sad
Guitar player in Lust 4 Thrust
Co-organizer of Python Serbia Conference
Co-organizer of BSidesNoviSad
Author of Freenit Framework
Author of Maolan

Presonus Troubles

Sat, 23 May 2015 22:00:00 +0000

For some time now, I have problems with Presonus AudioBox 1818VSL. The problem is the SPDIF input is muted. I tried this and that, I submitted an issue on Presonus ticketing system and nothing helped. First, Presonus, you’re morons. Every new driver needs to update firmware on the device, too, and there is no way you can update/downgrade driver without firmware or vice versa. Second, Presonus, you’re idiots. You’ve made USB compliant audio interface, only to change the mode to non-complaint in the newest firmware, rendering the device useless on GNU/Linux. Third, Presonus, you’re imbeciles. The controls you get with alsamixer is a joke. You can not mute/unmute digital inputs, for example. This is not true, and this was my problem for months now. Once Debian is going for reboot/shutdown, it saves the current state of audio interface to /var/lib/alsa/asound.state file. Every time Debian loads this file, digital inputs just stop working. And they so stop working, they don’t even work on Windows. What I had to do is uninstall driver on Windows, install new driver, reboot, launch Presonus’ mixer which says it has to update the firmware, fail, update firmware again, uninstall driver, install old driver (version 1.1, which has a firmware which is USB compliant) reboot, launch mixer, fail firmware upgrade (which is actually downgrade), update firmware and than it works.

At first, I said “OK, I’ll turn off Presonus before shutting down the PC”, but that’s not good. I have to think too much for one simple shutdown. Later on, I made /var/lib/alsa/asound.state link to /dev/null. That loaded garbage into the audio interface (so, go through Windows procedure from hell once again). Then I made /var/lib/alsa link to /dev/null. That solved the problem, but I didn’t like the solution. After countless days of poking, I realized that amixer returns interesting result for one of the controls:

amixer -c 1 get 'AudioBox 1818 VSL Clock Selector Capture Sw'
  Simple mixer control 'AudioBox 1818 VSL Clock Selector Capture Sw',0
  Capabilities: pswitch
  Playback channels: Front Left - Front Right - Rear Left - Rear Right - Front Center - Woofer - Side Left - Side Right - Rear Center - ? - ? - ? - ? - ? - ? - ?
  Mono:
  Front Left: Playback [on]
  Front Right: Playback [on]
  Rear Left: Playback [on]
  Rear Right: Playback [on]
  Front Center: Playback [on]
  Woofer: Playback [on]
  Side Left: Playback [on]
  Side Right: Playback [on]
  Rear Center: Playback [off]
  ?: Playback [off]
  ?: Playback [off]
  ?: Playback [off]
  ?: Playback [off]
  ?: Playback [off]
  ?: Playback [off]
  ?: Playback [off]

Wait a fucking minute!!! Something is off? Let’s make it on:

amixer -c 1 set 'AudioBox 1818 VSL Clock Selector Capture Sw' 'on,on,on,on,on,on,on,on,on,on,on,on,on,on,on,on'

Guess what? Every reboot works like a charm now. I have no idea why Presonus reports these to be off on the first run, but that’s the fix. If you have similar problems, I hope this is the solution. Next, record album, finally!

The Hell

Thu, 14 May 2015 22:00:00 +0000

Everyone hates the hospitals, I’m sure, but recent events made me wonder if I ever noticed how bad health care in my country is. It made me so sad to see the great clinic of Belgrade, the clinic where all worst cases are sent to, is in such a bad shape. Sad and eager to do something. I still don’t know what, but a hacker in me can’t sit still. The hard part is that my beloved one is in it right now. I can’t make up my mind if that’s a good or bad thing. We know she needs an operation for some time now, and that clinic is the best place in the country to do it, but I’ve seen how it looks. One of the terms that hits my mind a lot is really not nice. It’s 4:30 in the morning, and I can’t sleep. I will be hitting the road in today, and I can’t make myself sleep. I’ll be tired like hell, nervous, angry and all in all fucked up. Sometimes I wish I’m blind.

Hackerspace

Wed, 01 Apr 2015 22:00:00 +0000

I think it’s about 15 years now that I want a hackerspace in my town, and today that dream become reality. The hackerspace or makerspace (I never could tell the difference) is and will be alternate place for acquiring knowledge. Let’s go to the beginnings and see how it all unfolded.

Inspiration

I started dreaming about hackerspace more seriously about 2 years ago. What was the real trigger is the last year’s BalCCon. One of our guests was Mitch Altman. If you never heard of the guy, I recommend his TED talk in Brussels. The man is one of those smiley faces that radiates out “it’s easy if you really want it” message. One night we were sitting in the basement of the museum where the BalCCon was held and talked about starting a hackerspace in Novi Sad. He had a lot to say, but only one thing he said stuck in my head: “Out of 60k hackerspaces round the globe, not single one ever failed because of lack of funding”. It took me two weeks to process that. To be more precise, I live in a place and time where people are happy if they have any job. Let me repeat that, it’s important: “ANY JOB”. That’s the reason I couldn’t just accept the truth about hackerspace funding. Once I accepted that in my mind, I was ready for the the next step.

You might think that the next step would be opening the hackerspace, but no. I had to talk to a lot more people (on BalCCon, of course) who had experience with this matter. I’ve heard all kinds of squat stories, funding initiatives, creative ways of persuading landlords how to keep the rent low, and such, but one man was really standing out: Arnd from Technologia Incognita (sorry, Arnd, I don’t know your online identity). Although Mitch is inspiring, Arnd gave me tons of stories from his hackerspace experience that really pushed me forward. With all of these educational stories, I’ve set my path.

DIY

As always, we tried asking for resources from other organizations and foundations with no luck. At one point I was pissed of by my incompetence to start a hackerspace. All other people are able to do it, but not me. I took it really personal, so I sit down and realized one simple fact: I have my own flat, but if I didn’t have it, I would be forced to rent one, like most of my friends do, and it’s the same with the hackerspace. So I decided to change the client, and the minimal salary I would be satisfied with, and I made it. Don’t get me wrong, there are friends of mine that give moral and financial support, I was just sick of planing projects to fund a hackerspace (in short, begging), and I was determined to be able to fund it on my own, if everything else fails.

Result

In short, the result is Tilda Center. I’m sitting in the hackerspace of my town while writing this post with Internet connection provided by the kind people of NS Wireless. We have our own 67 square meters of space to gather, hack, talk, exchange knowledge and welcome new people. Come, visit us!

Life Together

Wed, 18 Mar 2015 22:00:00 +0000

I’m trying to keep my private life private, but this is just too weird and works too good to hide. For few years I’m living with my girlfriend and things are sometimes good, sometimes bad, as in any relationship (love or otherwise). To be honest, I got sick of bad times, because in most cases the reason for the situation to become bad is ridiculous: “you forgot to…”, “you didn’t…” and so on.

It’s not that I never lived with someone, it’s just that I’m the only child of the only child, so you might think of how much work I didn’t do. I don’t know how is it to live with someone, but I do know something else: Scrum! You might find it weird, but it works. My girlfriend and I are having daily scrum meetings, sync calendars and plan the day. My idea was, if I know how to lead the project, and don’t know how to have a life with someone, then pretend it’s a project. It can’t be worse than this. :o)

Responsive sites

Tue, 10 Mar 2015 22:00:00 +0000

Probably this mother fucking web site sums it all up. With colorful vocabulary and down to earth arguments, it makes a good point. What my experience showed me is that designers are so rarely starting with “function first, looks second”. When I say “function” I mean make it user friendly. I think the fancy term for it is “user experience”. Once I stumbled upon a page explaining what user experience really is. Unfortunately, I can not find the link to it now. One of the examples was “don’t ask for a type of credit card”. When you read about it, Visa cards start with digit 4, MasterCard start with 51 or 53. There were tons of suggestions like that, but basically every site should ask for as less interaction with the user as possible.

To conclude with the quote from the mother fucking web site: “Good design is as little design as possible”. Think about it!

Troubles with pull requests

Fri, 27 Feb 2015 22:00:00 +0000

Imagine two repos with a pull request from one to another. What you’re actually doing when accepting pull request is equivalent of git merge, so your HEAD will point to the merge commit upon accepting the pull request. Nothing new there, right? But what if you have hundreds of commits and you find a bug? Well, that sure thought you to merge more often, for start. If not, leave the Internet!

One thing I don’t like about merges is that git log is funky. We as humans can not think in jumps, which is the way computers and git commit parent work. We’re stranded in git log, and that’s all that our brain can really comprehend. We have to draw the tree and curves representing merges and whatnot. Now just imagine you have thousands of commits. To be honest, yeah, there is information which commit is merge and which is “normal”, you can filter them out or show only merges and filter by different criteria, but what we all really want is “tell me what’s wrong” button/command. Listing all those thousands of commits one after the other really doesn’t tell you much.

But what if you rebase, or cherry-pick one commit at a time? Then the first commit that breaks the code will be spotted right away. Yes, it’s more painful because some conflicts you’re having with rebase don’t show up when doing merge. The reason to do it is that you really don’t want to think in jumps, which merge commits really are. Imagine continuous stream of commits (read: git log) which you don’t have to imagine in your head as trees and branches. No double parent commits, only pure code, one commit after another. It would be much easier to read, but to be honest, I have no idea how would someone implement it. As it is, merge commits (which pull requests really are) are the least evil we have.

Modern WEB applications

Sun, 22 Feb 2015 22:00:00 +0000

Who am I to tell you about WEB applications? Even worse, modern ones? I am system administrator, right? Well, not exactly. The reason I never liked WEB programming is that I could never see the point of trying to shoot HTML through the WEB server, load balancer and browser to tons of different devices with who knows what resolution and ratio. How can you even dream of supporting the jungle of devices capable of running a browser? Deep down I always felt that’s wrong. I never had an alternative, so I kept my mouth shut, but admit it, it’s a design level flaw.

That’s a few years old story of mine. Now I think different. With frameworks like AngularJS and CanJS, to name the few, it all changed. First, frontend is finally in the front. I mean, what frontend is capable of doing now is calculate the height and width of the elements based on resolution, because the code is running where it’s most suitable for this kind of calculations: the browser. Yes, I know you could alter the page with javascript before single page application frameworks saw the light of the day, but it was hackish and ugly.

One more thing changed. Backend became “only” the REST API, or the fancy word “DB with some minimal code”. If your application is only changing the models in the backend, you might look at the whole backend as the DB with the REST interface. That means no more DB procedures, you’ve got a proper scripting language at your disposal with multiple supported DB types, portable code and properly decoupled code and the DB. Single page application frameworks didn’t bring this to the table, but modern WEB app design and REST did.

While we’re at it, REST brought some optimization, too. Well, JSON did it, but it’s almost the same thing, as almost all REST implementations use JSON as a format. As only data that is really necessary to fill the template is transmitted.

One thing I always hated about WEB applications is authentication. Session? Cookie? That’s just plain and simple wrong. In the age when every application is behind a load balancer, how do you balance authentication? One solution is saving the session in the DB. Just think about it for a second. You’ll be saving a temporary information in a DB, which saves the data forever. The modern way of dealing with this is forgetting the session and the cookies, and using JSON WEB Tokens, or JWT for short. Simplified, you obtain the token by POSTing to a authentication endpoint, that token is transfered in a HTTP header on every request, thus allowing the backend to know who you are. The token itself is just a very long string. It does have the structure and rules how to generate one, but I’m not going into that now, as it’s too much detail for this post. If all your backend servers generate and use the token in the same way, there’s no difference which server actually generated the token, which enables you to have proper load balancing.

This is just my point of view. It will evolve over time, I’m sure. You may find I made mistakes in my judgment. I’ll admit it right away, I’m not experienced in this field as much as I’d like, but I still do know the principles.

CCC Summary

Thu, 01 Jan 2015 22:00:00 +0000

To be honest, this CCC was mental healing for me. I went on a trip with a lot on my mind and I just couldn’t relax. I didn’t relax until the last day of the congress. Why? For 8.5 years of my professional carrier as a system administrator, I’ve never heard “good job, man” in as many words as “what did you fuck up?” kind of critics, so it finally got to me. On the other hand, CCC is all about sharing knowledge and telling interesting stories to other people. It relaxed me because I realized that, although I’m totally weird for people in my country, I’m nothing strange on the conference. I was accepted. FINALLY!!!

Let’s start from the beginning. My fiancé was so excited about CCC. I still can’t understand her, as she’s something like a child psychologist. There are topics covering e-learning and stuff like that, but she understands hacking, programming, administration, security, privacy, etc. So, for the first time in my life I’m with the girl I don’t have to explain what my job and hobbies are all about. What a relief! Second, she booked us tickets for hacker tour of DESY, particle accelerator. What? My girlfriend not only understands me, she’s pushing me in the right direction? I must be dreaming. Let me repeat. My fiancé was excited about seeing particle accelerator more then buying new camera (she’s also amateur photographer). How cool is that?

OK, we’re at the airport. As I’ve never flew in an airplane before, I had no idea what to expect. It was fun, of course. Seeing the clouds from above, seeing the Sun, as it’s winter and it’s cloudy every day. The G force when taking off was something I should have anticipated, but I was so surprised. And most of all, the size of Munich and Hamburg airports was astonishing. Let me explain. I was born and I still live in a town of 300k people. We don’t even have an airport, so seeing something that huge was great. Oh, and seeing that even Germans get some things too late was so precious to me.

We’re in Hamburg. As my fiancé and I can’t speak German, it’s fun. It’s so well organized that we can not comprehend it. It took me 4 days to figure out how to properly read S bahn maps. Germans are looking at us like Martians when we ask something like “in which direction is Dammtor station?”, because it’s so obvious to them. Let me tell you, I don’t mind. I come from a different culture, if you can call it a culture at all, and having diversity in almost everything is something I find useful. Of course, you feel like a jerk at first, but it’s an adventure.

We’re at the Congress Centrum Hamburg, finally. It’s so huge I didn’t get to know my way around it in 4 days. I still couldn’t find my way around, and we heard from an older W Holand member that they maybe return CCC to Berlin, but to a proper congress center big enough for 50k people. I was like “What? That’s 1/6 of the population of my town! Do you rent a bike to go around it?”. Anyway, they have few solutions in mind when CCH becomes too crowded (it’s already a bit crowded). You can download all lectures from CCC FTP. To be honest, I can point at only one lecturer, a friend of ours from Varaždin, Croatia: UNHash - Methods for better password cracking.

After congress, we had new year eve celebration at our friends house from Hamburg who speaks really good Serbian, so we finally could forget English. He was super nice and prepared beer and dinner I will never forget. Even if it was the worst night ever I would still have to say it was the best ever, because of his kick boxing black belt. No kidding with this guy. :o)

For the end, I have few things anyone traveling to Germany and not knowing German language should know. First, Turkish food is THE BEST! Sorry folks from other countries, but Turks prepare insanely large and delicious dishes. You can’t compete with that. Second is, if you want to ask a random dude on the street for directions or help in English, ask the black guy, as Germans and English are not really tight. It’s not about racism, it’s just that there’s a huge chance that black guy is actually from USA and can understand you. Of course, we decided to take German classes once we get home. Ideally, knowing English, German, Spanish and Chinese, you’re prepared for the world. We’ll see how good we’ll become at learning new languages, but hey, I learn new programming languages once a year. What’s a new spoken language compared to that. :o) We’re preparing for the flight home, as tomorrow morning we’re departing. It was nice, useful, helpful and fun. Thank you for all the fish!

CCC

Mon, 29 Dec 2014 22:00:00 +0000

Of course, I’m talking about CCC. As there’s a lot of talk about hacking this and that on a technical level, I will try to avoid that.

It’s the third day of the congress, and, as expected, I’ve seen all kinds of crazy lectures. The variety of topics is astonishing. For example, I’m currently sitting next to the ring making workshop with people hammering metal for hours. We’ve seen food hacking base, where you can learn how to cook like a hacker: using arduino to control temperature of kefir and such. You can see lectures like “copy wrong” dealing with weirdness of copyright laws and licensing (hint: you can take panorama pictures of Eiffel tower by day, but not by night because lights on it are made by architect that is not “dead + 100 years”). You can also leave your kid in the area with rubber balls and toys, so parents can watch their favorite lecturer. You can learn how to solder and work with arduino for TV B Gone. You can hear what’s the real problem with quantum computers where camera man asked some really technical questions, and the girl from the crowd knows so much more then I do! Don’t get me wrong, I don’t think it’s bad that girls know such things, but with my background of growing up in and old fashioned society, that’s weird, and I like it! And my favorite, creating your own audio devices and programming firmware and software for it (hint: software like pure data)

Let me also tell you that all the people are super nice. Everybody is collecting trash, either someone else’s or their own. Everybody is environment friendly, helping the neighbor or spreading awareness of all kinds of crazy cultural topics (hint: there’s a “medical advice: please wash your hands!”). For the end, do I need to tell you I didn’t expect this?

Presonus Audiobox 1818VLS 2

Mon, 15 Dec 2014 22:00:00 +0000

And it finally arrived. First, right channel on the headphones didn’t work. I made a peace with myself that I have to spend some more money for sending it to Thomman to replace it with a working one. I tried everything that came to my head and couldn’t figure out what’s wrong. As my friend brought it for me, we were not at my home, so on the way home I figured out that I didn’t check the pan. While unpacking it, there was a trillion of “please be the pan” prayers. I’ve connected it, fired it up, and everything was just working. I still have no idea how or why. And now for the technical details.

The reason I wanted this card is because it’s the only USB 2.0 audio interface that is seamlessly supported under GNU/Linux that I know of. You just plug it in, start JACK, and you have 18 inputs and 18 outputs. Just so you know, 1+2 outputs are the main, 7+8 are headphones and 9+10 are S/PDIF. As for inputs, 9+10 are S/PDIF and the rest is as it’s numerated. My setup is that in first input I have my guitar (1st and 2nd are mic/instrument combo, others are mic/line). The guitar is always routed to S/PDIF output, and that output goes to my guitar processor’s (Line 6 POD X3 Pro) input. S/PDIF output of POD is returned into S/PDIF input on the card. This setup enables me to record dry and wet guitar at the same time, as POD always gets it’s input from Presonus, I never have to change inputs or outputs with it, and I can reamp dry tracks. One of the problems I had with a previous setup was that audio card didn’t have guitar input, so I had to use POD for everything. This means, that while recording dry guitar, I would listen to analog output of POD with full simulation of amp for monitoring, and send dry signal through S/PDIF. Once I record it, I had to switch input to S/PDIF, which is the last item on the menu where you choose your input. Guess what’s the first option: guitar input, of course. So I spent a lot of time going back and forth through the menu which doesn’t go to first item once you’ve passed the last one. What a hassle! And that’s not enough. Because previous card, M-Audio Delta 1010LT can not be worldclock master, I had to switch the master every time I wanted to switch to “studio mode” or back to normal mode. Switching to studio means POD is the master, I switch to S/PDIF sync on Delta and everything works. The trouble is going back, because you can’t just switch the sync source. You have to stop JACK, switch the sync source, start JACK and start all programs that don’t handle JACK stopping well. If I just power off POD while it’s sync master, Delta just dies. One reboot later you’re good to go. That was a pain! Now I can even bring my studio anywhere and record. That means that I can finally have the same setup and latency no matter where I record (we used my guitar player’s laptop and audio interface for vocals).

Now for some fine tuning. First, every USB interface should have 3 as number of periods, compared to all other interfaces which have 2. You really want linux-image-lowlatency. My stable setup with a xrun now and then dropped from 10.5ms to 2.7ms. On 10.5ms you can notice the latency if you play something fast, and we do. Of course, I use maximum sampling frequency of 96k. I recommend using KXStudio repository, Cadence for JACK management, Ardour3 for recording and SoundCloud for sharing tunes. Hear ya soon!

Life is a Dream

Tue, 09 Dec 2014 22:00:00 +0000

Ever heard of it? Life is a dream? Did you ever think about it? I did. A lot! And I can tell you it is. Just look at what we do, create, transform and whatnot. Just by wishing something it comes true. Just by work you get results. When you think about it, you could do everything differently in your life. You could shape yourself, your life, your surrounding. Any of us could. I mean, even quantum physics says that anything is possible. What do we do with that? Dress up and go to boring work. Are we insane? Are we media oriented? Are we blind? In a word, fuck you, life. I publicly express my regret for existing in this world. I regret I’m incompetent to change the world (not really, but it involves starting from scratch). I’m sorry, but the mediocrity of the world got to me. Where do I sign up for a mission to Mars or exploration of black holes?

Presonus Audiobox 1818VLS

Sun, 07 Dec 2014 22:00:00 +0000

For years I have M-Audio Delta 1010LT. It’s great, but it lacks some things I need now. One of the things is being worldclock master. Second is Hi-Z input. All of that is covered with Presonus Audiobox 1818VLS. As a consequence, I can use my Line 6 POD X3 Pro without switching who’s the master on a card (it kills all audio programs, too) and without ever changing the routing or setup. What it basically means is that POD can run in stereo mode in which it expects two guitars on the inputs, and each guitar has only one channel (left or right). The reason I want this is so I can route from Ardour’s outputs of clean guitars to POD when doing playback, and route one Audiobox’s input to two inputs of POD, emulating two guitars. Why am I telling you this? I’ve ordered Presonus, and I can’t wait to hear how it works with my Ubuntu. It’s almost Christmas :o) Once it’s in my posession, I’ll tell you all about routing audio signals via JACK.

PCEngines APU

Tue, 02 Dec 2014 22:00:00 +0000

I had TP-Link provided by ISP for years, and I never bothered to replace it. It does work, but sometimes it’s just a pain in the ass. For example, the closer my laptop is to the router, more packet drops I have. I finally decided to grab a new router, and, of course, I got an “underground one”. I mean, who ever heard of PCEngines? Their model APU has variant with 4GB of RAM. Impressive, right? :o) There is no particular reason I decided for 4GB. It was there and I didn’t want to think what I can and can’t do. It’s loaded with 16GB SSD hard drive and Atheros WiFi card with 2 antennas. It has 3 gigabit ports and serial console. ALIX, former model had VGA, also, which made things much easier, because finding null terminated USB to RS232 cable is near to impossible (thank you KT). So, I grabbed pfSense USB image with serial console, dd’ed it to USB stick and install is pretty straight forward.

Let’s take a step back. You get all the pieces and it’s not tricky to assemble it if you follow instructions. Trust me, it’s not hard even the first time (I’ve never had anything similar before). For pfSense installation, follow [Gooze instructions](http://www.gooze.eu/howto/pfsense-installation-on-alix-apu-board-h owto) and you’re set. Next few tips are just to get you there easier.

First, APU’s serial port is on baud of 115200, and pfSense installation is on baud of 9600, so you boot on one baud, configure device, switch to lower baud, do the installation of pfSense and switch back to higher baud. Once you install the device, you don’t have to change baud ever again. Second “trick” is that in order to use WiFi card as AP, you have to assign ath0 interface as optional interface, then rename OPT1 to WIFI in order to find it easier in the future, and configure the same filter rules like those for LAN. That should be enough for you to start using WIFI. On software side, you only need minicom configured with /dev/ttyUSB0 on your laptop, and pfSense installation. Hardware: USB stick, APU, USB to RS232 converter and F2F (meaning “female to female”) serial cable. Have fun! :o)

Freelancing and USA company

Thu, 27 Nov 2014 22:00:00 +0000

Sorry, folks, but I’ve got nothing nice for this post. Last night I got an email that made me laugh and angry, at the same time. I tried to negotiate some freelance job, and after a week of talk I get “send me a Word copy of your resume and a copy of your drivers license or passport”. Right. Word? For Linux administrator? First big fail by the company. OK, I copy/paste my resume into LibreOffice, do some minor editing and send it with “My Word copy of a resume is attached … I can’t send one (copy of a passport, that is) to random people on the Internet just because they ask me to. Sorry”. Am I being I jerk? I don’t think so. Think about it. You can reset almost anything with a copy of a passport and email address. Almost anything, for example, was my password for a credit card site. I might be crazy, but I’m not dumb. What’s their excuse? “We’re going to give you root login, and we’re serious US company” (in much more words, so I’m paraphrasing). So that should give you credibility to ask people for passport? Right! Naturally, they canceled the job. To be honest, they just got to a job post first, otherwise I would do it.

Let’s go back a little. Let’s say I want to hire you and I say “I’m a representative of a US company”. Would you give me your passport? How do you know I do have a company? Internet site? Or I’m really registered. It’s easy to do that even in my shit hole of a country where you usually need tons of papers, but not for a company, they want to get your tax money as soon as possible. Just think what all you can do with someone’s ID or passport. Now add to that the fact that I’m a hacker, meaning I spend my time thinking about intrusion, malware, viruses, social engineering and who knows what else. Why? Because I’m usually hired to think about that and come up with a protection. In a word, I’m dealing with all kinds of lies and deceiving people, but all of a sudden, if you happen to have a USA company, I should trust you? I mean, why even emphasize your in the USA? World terrorist number one country in the world should give me some fuzzy feeling and I fall into hypnotic state and trust you completely? Get the fuck out!

And let me tell you, I did think this through. I didn’t want to say who they are, but fuck you retards from [Epic Farts](http =//epicforce.net/). If you mark me as “not smart enough” when declining my application, I can be jerk, too, you dumb ass.

Cloud Computing Basics 7

Fri, 21 Nov 2014 22:00:00 +0000

There’s been so much talk about Consul-template and I’ve never shown you any examples. Consul-template is what gives Docker container the knowledge when something is changed in Consul. It does it by having a template for configuration file filled with Consul data and command to run when template is generated. Let’s take a look at my Dockerfile for NginX:

FROM nginx:latest
MAINTAINER Goran Mekić <meka@lugons.org>

ENV DEBIAN_FRONTEND noninteractive
ADD consul /app/consul
ADD consul-template /usr/bin/consul-template
ADD run.sh /run.sh

CMD /run.sh

It’s a bit dirty as I get consul-template out of the sky, but kids, don’t be like me. :o) Yeah, always works!

I have my run.sh:

#!/bin/bash

set -e

rm /etc/nginx/conf.d/*
consul-template -config /app/consul/nginx.conf
sleep 1
tail -f /var/log/nginx.log

Great. So Consul-template does all the work. Let’s see the config.

consul = "172.17.42.1:8500"

template {
    source = "/app/consul/nginx.tmpl"
    destination = "/etc/nginx/conf.d/onelove.conf"
    command = "service nginx restart"
}

Remember I’ve told you that Consul is available on 172.17.42.1 on all hosts? This is where it comes handy. It is achieved through docker0 interface. Ensure you have the same IP on all hosts and that Consul is bind to it, and that’s it.

Second part is the template. What, where and how are the questions it answers. What source template I should use? Where to put the output? How to notify the service. Yeah, I’m aggressive, I restart instead of reload because of simplicity. The problem I had was with the run.sh. If you look closely, only thing I really call is consul-template. I never start nginx by hand or automatically inside the container, so if I have used reload, there would be nothing to reload initially (read: container doesn’t boot).

There are two thing you can do to ensure that your load balancers don’t reboot all at the same time. First one is to explore (wait parameter)[https://github.com/hashicorp/consul-template#usage] and restart with random delay. Second one is to initialize the config, start services, and watch for changes. When there is a change, reinitialize config and reload service. Example of it is One Love API.

#!/bin/bash

set -e

export COMMAND="consul-template -config /app/consul/api.conf"

echo -n "Waiting for initial config "
until $COMMAND -once; do
    echo -n "."
    sleep 3
done
echo " done"

uwsgi --ini /app/uwsgi.ini
python /app/manage.py migrate --noinput
python /app/manage.py collectstatic --noinput

$COMMAND &
sleep 1
tail -f /var/log/uwsgi.log

Now you know everything I know about the cloud computing. This is my way of doing things, and my view. You don’t have to agree on everything. I’m not going to agree with these posts in a year! What I didn’t describe is Docker HUB, but go to site, register, and add your repository. Play with it, it’s dead simple. I gave my best to give you posts that have no bull shit talk, because I was mad on todays media representing the cloud. I hope I did it. Happy dockering!

Real Time Feedback

Fri, 21 Nov 2014 22:00:00 +0000

As I’m learning about WEB development, I’m learning how not to do it. Anyone can do it, but only few can find so many solutions that are bad and know why they are bad. That means that those few will nag big time about most of the solutions, but once they shut up, you know you’re doing it right. I’m lazy. I know it’s my strength and my weakness. That’s the reason I tend to automate everything. The trouble is that I also tend to optimize a lot. I hate inefficient procedures. That’s my reason for two monitors. Think about it. I’m writing every post in vim. That means I have no idea how my post will look like. But if you take a look at the code, you’ll notice it is folded on 80th character. We’re all crazy in our own way, I just accepted my weirdness. :o)

OK, now the important stuff. When you write in vim, you realize you can execute a script on every save. What if you’re editing a post, the script you’re running remembers the window which has the focus, focuses chromium window, sends it “CTRL+r”, and returns back? Basically, you’ll have your post rendered every time you change the post. And what if you have two monitors? You could see the change almost while you’re typing. That’s pretty close to real time feedback. And if you have real time feedback, you can see what you’re doing, which is great. Now tell me, isn’t this the best reason you’ve ever heard for a monitor purchase? :o)

All you need is vim config like

au BufWritePost * silent !/home/meka/bin/vim-reload-chromium.sh

and the script

#!/bin/bash

exec 1>/dev/null
exec 2>/dev/null

ACTIVE_WINDOW=$(xdotool getactivewindow)
CHROMIUM_WINDOW=$(xdotool search --name '^.* - Chromium$')

sleep 2
xdotool windowactivate ${CHROMIUM_WINDOW}
xdotool key "CTRL+r"
xdotool windowactivate ${ACTIVE_WINDOW}

Now go, prepare for Christmas! :o)

Cloud Computing Basics 6

Thu, 20 Nov 2014 22:00:00 +0000

Docker here, Docker there, and no Docker build anywhere. That’s not fair. How about we create the first docker? Docker image is built from Dockerfile. The syntax of Dockerfile is almost the same as Bash. This is an example of Dockerfile:

FROM debian:jessie
MAINTAINER Goran Mekić <meka@lugons.org>
RUN touch /some-file.txt

To build it run docker build -t username/repo . in a directory containing Dockerfile. It’s advisable to have a HUB username, as registration is free and has autobuild capabilities. We’ll deal with simple builds for now, and leave autobuild for some future post.

To upload your image issue this command:

docker push username/repo

Docker will ask you for username/password/email combination. Fill it up, wait for upload to finish and that’s it. You have your first Docker image. Let’s give it a spin.

docker run --rm -i -t username/repo /bin/bash

What it does is (simplified):

--rm: remove container when it stops
-i: this will be interactive container (read: someone will type commands in it)
-t: give me a terminal emulation

Note that every command in Dockerfile will create additional layer. It means that Docker images are organized as multiple file system layers which have dependencies. Much like a git repo branch is pointer to commit which has its own dependencies, Docker image remembers which file system layer is on top. As every layer remembers which layer it depends on, you can have dependency line. In the example above, there will be at least 3 layers: FROM, MAINTAINER and RUN lines make them. This has consequences you have to be aware of. First, if the line in Dockerfile and dependent layers didn’t change from last build, Docker will use last build’s layer, not build it (read: cache). Second, EVERY line in Dockerfile creates layer. So, if you create 1GB file on one line, delete it on the other, you’ll have a small layer (from deleting 1GB) dependent on a big layer (where you created 1GB), although lower layer is unusable, because upper layer effectively masked it.

The “trick” I use is to have a build script which will cause 2 lines in Dockerfile: one for ADD and one for RUN. I start with debian:jessie, add all build tools, build my app, remove build tools and do the cleanup. The build does last much longer, but you end up with MUCH smaller images. I managed to shrink One Love API image from 1GB to 298MB just using this. What I think would be the optimal solution are two images, one for building, one for using application. And guess what. There are. For example, you have python:latest and python:onbuild images.

I leave it up to you how to build your applications, these are just some ideas. Idealy, you can base your application on busybox environment, and use images as small as 5MB. If not, lurk around for your perfect solution.

previous next

Cloud Computing Basics 5

Mon, 17 Nov 2014 22:00:00 +0000

You hear me talk about thousands of servers and never about how to get to the point where you make 10,000 servers go up. The fancy word is “provisioning”. I don’t even know what’s it supposed to mean, but for me it means “make a recipe which will make machine configured for a purpose”. As we’re talking about cloud computing and Docker, what we need is a machine that is Consul and Registrator ready. Getting to that point on multiple servers is not such a short task, but it’s repetitive. And, along comes Ansible. Over the years people realized that they want a language for specifying server configuration that is declarative, so Ansible uses YAML for that. We also need a language to describe the configuration of different services, andJinja2 offers that. On top of that, it’s Python, so it has extra plus on my scale. As you can’t just apply something on your servers and hope for the best, you need some development environment. I think Vagrant has no competition in that field, yet. Let’s see on the example of this blog how to use it. Clone meka.rs, and execute vagrant up. It should download CentOS 7 box, create new virtual machine in VirtualBox and provision it with Ansible. What it does is not that important right now. Let’s start with example Ansible task:

- name: install docker
  sudo: yes
  yum:
      pkg: docker
      state: latest

Yeah, it’s that simple and readable. That’s fine, but let’s dive into the details. First, there’s a site.yml file. You can see the title of Ansible playbook, hosts on which it will be run on and roles it will apply. As I have two roles but mention only one in site.yml, it means that role common is a dependency of meka. That dependency is noted in meta. There’s only one thing missing: list of servers that this playbook applies to. Vagrant will generate that list on the fly depending on the configuration inside Vagrantfile, like number of machines. Nice thing about Vagrant is that it will share directory of the repo as /vagrant inside virtual machine. Basically this means that you can code on your laptop/desktop with your favorite editor and see those changes on the same distribution that is used in production, on the same docker that is used in production. Idea is to have almost identical environment in all stages, like development, testing, pre-production, production, … It enables developers to make less bugs that are caused by differences in environments.

Ansible is such an important and easy to use tool, that I have no more words to describe it but “read all from the provision directory”. It’s simple and powerful.

previous next

Cloud Computing Basics 4

Sun, 16 Nov 2014 22:00:00 +0000

So, you’ve seen how Consul and Registrator can be combined in previous post. How about we get down and dirty, finally? One way to start a docker container is to run it with systemd. It’s my favorite way, so let’s take a look how does Consul service looks like:

[Unit]
Description=Consul
After=docker.service network.target
Requires=docker.service
Wants=network.target

[Service]
TimeoutStartSec=0
ExecStartPre=/usr/bin/docker pull progrium/consul
ExecStart=/usr/bin/docker run -h site -p 8500:8500 -p 53:53/udp --rm --name consul progrium/consul -server -bootstrap -advertise 192.168.33.33
ExecStop=/usr/bin/docker stop consul

[Install]
WantedBy=multi-user.target

There are small, almost hidden but important pieces of this code. First, you see there are multiple ports published, and that is:

8500: HTTP
53: DNS

There are at least 4 more ports exposed on consul Docker image, but this is more than enough. In previous posts you could read that Consul’s HTTP interface has REST API and UI. If you visit Vagrant IP, you’ll see all the services that are registered. That’s the port that Registrator uses to do it’s magic. But more interesting port is 53. First, notice it’s UDP, not TCP. Second, that port is DNS. In other words, if you run container which publishes port 80, Registrator will pick it up and register it in Consul. That means that you can ask Consul’s DNS where is your new created container like this

dig @172.17.42.1 web.service.consul

IP of 172.17.42.1 is interesting because it’s IP of docker0 interface which is available on every host. As Consul is distributed among all hosts, and that IP is available on every host, you can tell all containers to use 172.17.42.1 as DNS:

[Unit]
Description=PostgreSQL
After=registrator.service network.target
Requires=registrator.service
Wants=network.target

[Service]
TimeoutStartSec=0
ExecStartPre=-/bin/mkdir -p /var/lib/docker/volumes/postgresql
ExecStartPre=/usr/bin/docker pull paintedfox/postgresql:latest
ExecStart=/usr/bin/docker run --dns 172.17.42.1 -P -e SERVICE_TAGS=master -e DB=onelove -e PASS=password -v /var/lib/docker/volumes/postgresql:/data --rm --name postgresql paintedfox/postgresql
ExecStop=/usr/bin/docker stop postgresql

[Install]
WantedBy=multi-user.target

Notice --dns option. Now, this is the flow of DNS data/queries. Every container will ask Consul for .consul domains. If FQDN container is asking for is not subdomain of .consul, Consul will ask external DNS, which is usually 8.8.8.8. In the previous example, I’ve set a service tag to master. Because it is a service for PostgreSQL, which might be part of DB cluster, you must have a master server. Although there’s only one DB server here, I still like to set master just in case I decide to scale later. On DNS side it means you’ll get master.postgresql.service.consul records. As a matter of fact, you’ll get two records: A and SRV. First one will only return IP address, while second has richer structure which includes IP and port. So, if your application depends on DNS only, you can still dockerize it. Nice thing is that queries are super fast and are not cached.

For the last, one trick I use lately. My DB host is always master.postgresql.service.consul and I don’t even generate it on change with Consul-template. Fact is that when DNS records change, as DNS is not cached, application server will hit new DB server the second it’s in DNS. For some other neat tricks, check out Consul documentation.

previous next

Cloud Computing Basics 3

Sat, 15 Nov 2014 22:00:00 +0000

So I’ve scratched the surface of a service discovery in a previous post. Let’s dream on. Up to now, I’ve only written about running a single container on a single host. But I’m poor, I want multiple things on the same hardware. Let’s go back a little. Docker containers are like tiny virtual machines. They have IP address, disk space, running processes, … Strictly speaking, docker isn’t a virtual machine, but it almost is. There’s one quirk about containers: they can open PostgreSQL port on a random port. Here’s the reasoning. If you want to host multiple PostgreSQL instances, they can’t all be on the same port. “Not a problem”, one would say, “I’ll configure different PostgreSQL instances on different ports”. There’s only one problem: you’re running all PostgreSQL containers from the same image. Yes, you can put configuration at runtime, but remember, we have 10,000 hosts. You just can’t do it efficiently. OK, this is how Docker does it. PostgreSQL is, by default, listening on port 5432. If you just do docker run postgres, you won’t even open a port. That’s Docker’s weirdness about ports. Docker images are built with predefined ports to be opened in a container. But that same port doesn’t get mapped unless you explicitly say so with -p or -P. Here’s an example

docker run -p 5432:5432 --rm postgres

What it does is tell Docker to bind container’s port 5432 to all interfaces on the host on the port 5432. In one word, you have your 5432 port open on the host. You can do something like -p 127.0.0.1:5432:5432 to listen on localhost only. You can, also, do things like -p 127.0.0.1::5432 and Docker will bind container’s 5432 port to a random port on localhost. But how do you know where to find it? You can use docker port <container> 5432 and it will tell you. I agree it’s not the most elegant way of figuring out a port, but it works. You’re able to put multiple docker images with the same open port, and they will be happy to run together. Also, you will probably want to use private network IP instead of 127.0.0.1 and have all your hosts on that network.

If your containers are secure enough, and they never are, you can expose all ports on all interfaces. This means that every port is accessible on any network interface of the host. To do that

docker run -P --rm postgres

The docker port command will still work and everything else remains the same.

By now you probably wonder how Docker knows which ports are open on the container. No, it doesn’t seek all open ports. You must specify which ports will be opened (or exposed, in Docker terminology). That’s part of the build process which I will cover in some later post. For now, just remember that one line of Dockerfile for PostgreSQL is

EXPOSE 5432

Two old problems remain:

how to run it in 10,000 servers environment
how to tell application container the DB info

No wonder, it has something to do with Consul. One really neat container is Registrator. It listens for Docker events and every time new container runs, it registers container’s exposed ports to Consul. After that, let Consul-template do the rest.

I’m starting to go in “WTF” direction, again. I have a proof this is not an empty story. If you clone One Love and follow the instructions on that page, you’ll get my project (in early alpha at the time of writing this post) which utilizes all I was talking about. It takes a fair amount of time for it to download everything, but once it does, you have my application in virtual machine, but that’s not the reason I told you to do this. The reason is that you have Consul’s WEB interface available at Vagrant VM. Now go and play with it :o)

previous next

Cloud Computing Basics 1

Thu, 13 Nov 2014 22:00:00 +0000

The first critic I’ve got for a blog was “Yeah, it’s all nice and shinny, but WTF? Where are examples, documentation, … ? How do I do it?” Let’s go back a little. Cloud is extremely complicated network of hardware and virtual machines where one second your server is in New York, next second is in Amsterdam, if you happen to believe the media and PR “experts”. In reality, yes, sometimes it might happen that your machine gets migrated to another data center, but why would anyone do it if there’s no need? What cloud provides is easy migration of machines, it doesn’t enforce them! In other words, just because you can scratch your nose, doesn’t mean you have to scratch it all the time, but when you do, it’s better if you’re able to do it in seconds. That’s all there is about the cloud. The logic is simple. If you can have a number of machines that are basically junk, you can create highly redundant WEB site if all machines are on the fast network. You don’t mirror all the data between them, but download virtual machines or docker images fast when you need it and hope nobody noticed.

That’s where Docker comes in. Docker images are made to be as small as possible. For example, Consul image is ~50MB, Registrator image is ~20MB. Of course, not all images can be that small, and I did mention two of the smallest images I know of, but maybe your application can sit on top of it (base image is called busybox). Or you might need big image only to build your application, but not to run it? Docker optimization and security are separate and hot topics these days, and I’m trying to give you the basics here :o)

What marketing around cloud technologies will tell you is that “On hardware level we have multiple machines working as a unit to run your code on, so if one fails, some other takes over”. Right. Let’s say you have two machines with docker. One is your main site, and the other one is just a spare. Main one dies. And what then? How does the spare becomes the main? Let’s say it’s a simple modification of DNS record. OK, we got our new IP registered. But we had docker container running on the old main computer only. Docker Hub is a service which offers hosting docker images, so your spare machine can, in case of this blog, run this:

docker pull mekanix/blog

Docker will do the magic of downloading and updating the image, if you had an old one, for you. As a matter of fact, why don’t you try it? Clone the repository of this blog, install Vagrant and VirtualBox, and run:

git checkout no-ansible
vagrant up
vagrant ssh

First line is there just to make sure you don’t need Ansible, yet. I’ll cover Ansible, provisioning and deployment in some of the later posts. To setup a docker:

sudo yum install docker
sudo gpasswd -a vagrant docker
sudo systemctl enable docker
sudo systemctl start docker

Your docker is now running, and you can control it. You control the cloud! To pick up the changes on vagrant account you have to logout and run the container:

vagrant ssh
docker pull mekanix/blog
docker run -p 80:80 --rm mekanix/blog

And that’s it. Go to Vagrant IP and you should see this blog. If you want to update the image, stop the container (CTRL+C) and

docker pull docker run

Happy docking :o)

Cloud Computing Basics 2

Wed, 12 Nov 2014 22:00:00 +0000

So I’ve scratched the surface of a cloud in a previous post. Let’s dream on. Let’s say you’re hosting a Django site. For start, let’s assume that everything is inside one big container: Django, PostgreSQL, NginX, …

After few months, your site grows and one machine is not enough. How do you add another, when your docker image has all included, and you should share the DB? So, you decide to have PostgreSQL in a separate docker, or someone is hosting it for you. The only thing you need from Django perspective are DB credentials. You end up with working Django cluster, but how do you register it in DNS? DNS is slow. If you kill one machine and start another, with different IP, DNS can take days to propagate.

So you decide to put a load balancer in front of those Django machines, because load balancer reconfiguration and restart can take 5 minutes or less if you know what to do. Much faster than DNS propagation. But now, you have a problem. If that load balancer fails, you don’t have a site. So you start yet another load balancer and have it proxy the requests for two Django machines, so if one load balancer fails, there’s another.

Over the following months your company and app grow, and you find your self in a position where you need 5 application servers, 3 DB servers and 2 load balancers. Imagine you have to change the password. You would do it on all 5 servers by hand, restating each when done. Or you’ve already discovered Ansible (or Puppet, or Chef, or SaltStack, or …) and do it automatically. But imagine you have 10,000 servers. It would take an hour, I suppose, to update all the machines. That means, when you change DB password, it will take you an hour to get to the last machine. That hour the last machine uses old DB password, which renders it non useful. It would be nice if machines could agree on a set of parameters that are needed and distributed. In our case, it would be nice if every machine could remember DB user/pass combination, so wherever you start you Django docker, it knows how to connect to DB. That’s exactly what Consul does.

Consul is an application with integrated WEB UI, REST API, DNS, RPC and tons of other things, but interesting for us is it’s ability to cluster and remember settings. I start it as a docker container on every host of the fleet. If you have 6 machines with clustered Consul, changing variable on one host makes it mirror to all other machines. That’s great, but it doesn’t tackle with docker containers, so Consul alone is not enough for your 10,000 machines data center. Consul-template is small utility which sits in your Django container and connects to Consul. When Consul variable changes, consul-template will generate the configuration from a template you provide and Consul data, and restart your Django. To be technically correct, it will execute command you configured it with, but you want that command to do the restart of you application once the configuration is generated. Now DB user/pass changes can be effective in seconds instead of hours.

Why stop there? You can have your DB docker react on changing the DB user/pass combination in Consul. In other words, if you add DB user to Consul, DB container will create it (if you “restart” consul-template script is smart enough). You can go wild with Consul variables and cloud setup. It’s up to you and your team to find what should be shared through Consul, and what should be part of on-disk configuration.

previous next

Service Discovery

Sun, 09 Nov 2014 22:00:00 +0000

Yes, docker is my new favorite toy, and it is big word on the Internet. So I have my hands in the gutter trying to figure out the optimal way to configure my cloud. What’s the big deal? For start, etcd is not my best friend, any more. I like Consul implementation and features much more. Hence, CoreOS is not the perfect distribution. I’m using Debian Jessie which is in beta2 stage in the time of writing this post. The reson for this switch is Registrator. Consul, Registrator and Consul-Template are 3 projects that make service discovery a piece of cake.

Consul is something like a distributed database. It stores key/value pairs. It sounds like “not much” but it’s built on top of Serf, so it’s built to be distributed. Consul, also, has built in DNS server, so key/value pairs can be A or SRV records. In one word “where is my postgresql docker container”.

Registrator is just crazy. You give it Docker’s socket file to listen for events, and every time new docker container is up, it automaticly registares them in Consul, and deregisters them when container is down. This means that every time new application container is up, you can query it via Consul’s DNS protocol. If you start your containers with -P, for example, Registrator will register the ports your container is bind to.

Consul-Template is golang program that listens for changes in Consul, generates configuration of a service (say, nginx), and runs a command (say, service nginx restart). Why is this cool? Just imagine you have your load balancer, application server and database server running. You have a peak, and you start one more application server, which needs to be added to load balancer, and configured with database connection. With consul-template, it’s as easy as starting new application container. Because Registrator will pick up on which IP and port it is awailable, consul-template on load balancer is able to regenerate configuration and restart nginx. If hosting provider has autoscaling support (I know AWS has it), you have your cloud dynamicly scaling. How cool is that?

Cloud Computing - intro

Sun, 02 Nov 2014 22:00:00 +0000

As nobody understands the cloud, it became obvious to me I will hack something out of it. And I did. A job. My latest toy is CoreOS. It features two nice things a man doesn’t need until he faces big amount of servers. First being collective consciousness, aka etcd, allowing storing information about the services, like which upstream servers for your load balancers are on which IP addresses. Second is fleetd, which uses etcd to store and read data. It starts docker containers on your CoreOS instances. Actually, it starts systemd services, which is even better! :o) Oh, did I mention cloud computing these days is all about running containers? Docker containers in my case. Lastly, in dynamic environment of the cloud, you need dynamic configuration. But what application in the world has dynamic configuration? None that I know. Despite that fact, cloud computing and docker containers are thriving. There’s a secret ingredient: confd. What it does is poll etcd every 10 seconds or so, checking if there are changes on some of the keys in it and if there is, generates the configuration of your application and restarts it. Confd uses templates and etcd data to generate the configuration, so once you have your fleet running, changing master DB host is as easy as

etcdctl set /my/app/db/server 192.168.0.4

and all containers having confd will pick up the changes. Just imagine, one command and thousand services change their configuration accordingly. This is small step for cloud, but huge step for a man, because it’s hard to start thinking about “configuration is in the cloud”. Once you grasp that idea, you’re on a great way to cloud computing.

Goran Mekić

Neural and Impulse Blending

Impulse Response Sweep

Related posts

Impulse Response

Dirac Pulse

Capturing Impulse Response

Related posts

Neural Amp Modeling

Re-amping

Preamp and NAM

Tips

Update

Related posts

Rust and ioctl

New Guitar

Gibson Les Paul Custom

Signature model

My impressions

Ratamahatta

OpenLDAP

DIY Sound

FreeBSD PF and Dummynet

Update

FreeBSD Dual Stack Firewall

Host

Jail

Router

Reggae

FreeBSD and YubiKey

FreeBSD Linuxulator

FreeBSD Dual Stack Jails

Jail Setup

Unique Local Addresses

Global Unicast Address

Global Unicast Address Behind Router

FreeBSD Dual Stack Network

Router

Laptop

Server

FreeBSD and IPv6 Tunnel

Router

Desktop/Laptop

FreeBSD Audio

Ne mogu vise da ćutim

FreeBSD Upgrade Bootloader

Serbia vs Internet

Freenit Framework

OpenLDAP Multimaster

FreeBSD Crossbow

CBSD Base Upgrade

FreeBSD Upgrade

FreeBSD Update

Videos

DaFED #61 - Kako automatizacija procesa pomaže u razvoju sigurnijih aplikacija

FOSDEM 2023 - Reggae: cool way of managing jails/VMs on FreeBSD

EuroBSD 2022 - FreeBSD audio in amateur and professional setups

FOSDEM 2021 - Managing virtual resources with CBSD, and beyond

BSidesLjubljana 2019 - How Did We Teach (Ourselves) Security

FOSDEM 2019 - FreeBSD in Audio Studio

BSidesLjubljana 2018 - Modern And Secure Devops on FreeBSD

FOSDEM 2018 - CBSD Isolation Manager

BalCCon 2016 - FLOSS Way of Funding a Hackerspace

BalCCon 2015 - What is Hackerspace

IT Feels Like Constant Failure

HardenedBSD Install Over Serial

CBSD Reggae

Getting Started with Digital Audio

FreeBSD Cloud and DevOps 3

NuttX and Clang

FreeBSD and USB MIDI

Arduino Due and Clang

FreeBSD Cloud and DevOps 2

Sing, beastie, sing!

FreeBSD Cloud and DevOps 1

Lost Passport

FreeBSD WiFi and Ethernet Bridging and Aggregation

Router Configuration

Laptop Configuration

FreeBSD Network Optimization and CCC Tickets