Jekyll2022-12-20T17:15:54+00:00https://jboulineau.github.io/feed.xmlJon BoulineauThe BlogArchitecture as Code: Managing Documentation2021-02-19T00:00:00+00:002021-02-19T00:00:00+00:00https://jboulineau.github.io/blog/architecture/architecture-as-code<p>For the past 18 months or so my developer/architect/leadership role mix has shifted heavily to architecture/leadership and that has lead to a significant increase in my need to manage documentation. Of course, I’ve been doing architecture for many years now and I’ve developed various ways for keeping it all straight, but it always felt clumsy. For instance, I do very much like Lucidchart for diagrams, but I do run into impedance in some areas:</p>
<ul>
<li>Not everyone with whom you need to collaborate has Lucidchart access.</li>
<li>Lucidchart diagrams often need to be be exported to other formats to be shared or included in documents.</li>
<li>Vendor lock-in. My company was acquired and they do not use Lucidchart, which means I’m going to have to convert to something else, anyway.</li>
<li>It is too easy to accidentally grant someone edit permissions.</li>
<li>Versioning is manual (named versions) or much too granular.</li>
<li>Reviewing changes between versions isn’t easy.</li>
</ul>
<p>I’ve set out on a number of occasions to look for a better way, but I’ve never found a tool chain that made sense. Here’s the tool chain that seems to be working:</p>
<ul>
<li><a href="https://diagrams.net">diagrams.net</a> for diagraming</li>
<li>Markdown + <a href="https://pandoc.org">pandoc</a> for documentation</li>
<li>git (specifically, GitLab) for version control and collaboration</li>
<li>Visual Studio Code</li>
<li><a href="https://plantuml.org">PlantUML</a> (maybe)</li>
</ul>
<h2 id="diagramsnet">diagrams.net</h2>
<p><a href="https://diagrams.net">Diagrams.net</a> allows persisting your diagrams as local files. I use the desktop app since it makes more sense in this model. The primary magic is that you can use a <code class="language-plaintext highlighter-rouge">.png</code> or <code class="language-plaintext highlighter-rouge">.svg</code> compatible format to store your diagrams, which means there’s no need to export the diagram to other formats to share with anyone who doesn’t use the same tool. And if you need to share the diagram between other tools (like lucidchart), simply convert the file to <code class="language-plaintext highlighter-rouge">.drawio</code> and it becomes portable in an editable format. I’ve found the <code class="language-plaintext highlighter-rouge">.png</code> format to work best, even though it suffers in visual quality somewhat. <code class="language-plaintext highlighter-rouge">.svg</code> compatibility is tenuous in Word (for instance) and non-technical users don’t grok them. And (at least in GitLab) you can’t compare changes between commits.</p>
<p>Being natively stored in an image format also allows them to be linked by reference into Markdown files so diagram changes are automagically reflected in my documents. This saves a lot of annoyance when I’m in the later stages of documentation and I’m making minor tweaks to diagrams to reconcile with my documentation.</p>
<p>This also allows for easy sharing. If I ever have to just drop a diagram as a one-off request to someone, I can just send the file, rather than export into something consumable. This is extra helpful for getting something to a VP who is putting together a Power Point and needs a visual.</p>
<h2 id="markdown--pandoc">Markdown + pandoc</h2>
<p>Everyone loves Markdown, right? It’s everywhere, presentable by the major documentation systems, convertable to everything, and very easy to do (almost) all the formatting you want to do. But, sometimes you have to share documents in other formats. For instance, for ‘official’ recognition I have to submit some documents in Word format. Luckily, there is <a href="https://pandoc.org">pandoc</a> and with little effort I can create the Word doc from the Markdown. You can even use a reference file to make the output conform to a required template. It’s not perfect, but it works well enough with some tweaking.</p>
<p>Other than the simple pleasure of just editing text rather than some proprietary format with all the button clicking, linking out to the diagram files rather than embedding them into documents has been the big win.</p>
<h2 id="git">git</h2>
<p>It’s ubiquitous and it aligns with how the teams I support work. For collaboration and versioning it can’t be beat, since almost everyone understands the flow now. It’s pretty cool to be able to compare changes to diagrams when working together and it’s very handy to be able to easily keep drafts of diagrams separate from published diagrams by using branches.</p>
<p>I’ve settled on a single repo with directories for every product I work on, but do what works for you.</p>
<h2 id="visual-studio-code">Visual Studio Code</h2>
<p>Like many others, <a href="https://code.visualstudio.com/">VS Code</a> has become my primary tool for everything. As usual, there are extensions that make all of the above (nearly) seamless.</p>
<ul>
<li><a href="https://marketplace.visualstudio.com/items?itemName=hediet.vscode-drawio">Draw.io integration</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=nopeslide.vscode-drawio-plugin-mermaid">Mermaid for draw.io</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=vitaliymaz.vscode-svg-previewer">SVG Previewer</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=DougFinke.vscode-pandoc">vscode-pandoc</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one">Markdown All in One</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint">Markdown Lint</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=shd101wyy.markdown-preview-enhanced">Markdown Preview Enhanced</a></li>
</ul>
<p>Being able to edit diagrams and documents, as well as maintain the repo, in the same tool is very nice.</p>
<h2 id="plantuml">PlantUML</h2>
<p>I left this one last because I’m very much on the fence. I really, really like the idea of diagrams-as-code, and <a href="https://plantuml.com">PlantUML</a> seems to be the most mature. At this point I use it anytime I need to do a UML diagram (but it can sort of do other diagrams, too).</p>
<p>I’m keeping my eye on this space as <a href="https://mermaid-js.github.io/">Mermaid</a> seems to have more attention and is more flexible (and prettier). Mermaid is supported in <code class="language-plaintext highlighter-rouge">diagrams.net</code>. There’s also <a href="https://diagrams.mingrammer.com/">Diagrams</a>, which looks really cool. I’ll play with it next.</p>
<p>If you are keen on using PlantUML, here are the extensions I’ve found useful:</p>
<ul>
<li><a href="https://marketplace.visualstudio.com/items?itemName=jebbs.plantuml">PlantUML</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=Yog.yog-plantuml-highlight">PlantUML Syntax Highlighting</a></li>
</ul>
<p>If you do use PlantUML with these extensions, you’ll need to structure your repo in a way that makes the build output create in a sane way, as output follows the structure of the directory containing your source files. This pretty tightly constrains how you set up your repo directory structure.</p>
<h2 id="what-i-dont-like">What I don’t like</h2>
<p>Tools like Confluence are definitely nicer to view documentation in than Git(Lab/Hub). I’m hoping to play with the git plugin for Confluence to see how it all meshes together.</p>
<p>Searching documents is a problem. Sure, many people I work with can just pull the repo and <code class="language-plaintext highlighter-rouge">grep</code> for what they want, but not everyone can. However, those people tend to just ask me rather than go looking. I’m toying with the idea of creating a Jekyll site since it works nicely with Markdown files and can enable search. For now, I think keeping the repo organized in a sane way is good enough.</p>
<p>Lucidchart diagrams look better and work better (but not dramatically so).</p>I've been thinking about using a git repo for managing my architecture diagrams and documents and I finally figured out something worth trying.Unable to connect to Kafka with kafkacat over SSL/TLS2020-05-26T00:00:00+00:002020-05-26T00:00:00+00:00https://jboulineau.github.io/blog/kafka/kafka-tls-issue<p>Occasionally it is helpful to use <a href="https://github.com/edenhill/kafkacat">kafkacat</a>, the <a href="https://github.com/edenhill/librdkafka">librdkafka</a> based CLI tool, to inspect our Kafka brokers. As our Kafka cluster runs in Docker Swarm, it is isolated from the rest of our network. Only containers that are in the the same Swarm virtual network can connect. As a result, I have an administrative container operating as a jump server to which we can connect to administer the cluster as needed. When kafkacat was suddenly unable to connect to the brokers after working perfectly fine for months, it presented quite a mystery.</p>
<p>After checking the basics such as network connectivity to the brokers, I was able to track down the issue to the TLS listeners, as kafkacat was able to connect via the plaintext listeners which are temporarily running for backwards compatibility. The brokers were also able to communicate over the TLS listeners, which deepened the mystery. I used our docker-compose files to locally create a replica of our cluster and was able to connect to it via kafkacat and the Python client, which also uses librdkafaka. Why, then, was this problem occurring from the administrative container?</p>
<p>Comparing to my MacBook Pro configuration I discovered a difference in librdkafka and kafkacat version that limited the TLS configuration options from the container. Kafkacat was also compiled with an old version of librdkafka while my laptop was on version 1.3.0. Reconciling these was harder than anticipated, as even the official repo for kafkacat had version 1.5.0 built with the older version of librdkafka. Finally, I ended up building both from source as can be seen in this Dockerfile excerpt:</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">RUN </span>git clone <span class="nt">--depth</span> 1 <span class="nt">--single-branch</span> <span class="nt">--branch</span> v1.3.0 https://github.com/edenhill/librdkafka.git
<span class="k">WORKDIR</span><span class="s"> /librdkafka</span>
<span class="k">RUN </span>./configure <span class="nt">--install-deps</span> <span class="o">&&</span> make <span class="o">&&</span> make <span class="nb">install</span>
<span class="k">WORKDIR</span><span class="s"> /</span>
<span class="k">RUN </span>git clone <span class="nt">--depth</span> 1 <span class="nt">--single-branch</span> <span class="nt">--branch</span> 1.5.0 https://github.com/edenhill/kafkacat.git
<span class="k">WORKDIR</span><span class="s"> /kafkacat</span>
<span class="k">RUN </span>./configure <span class="o">&&</span> make
</code></pre></div></div>
<p>Using the option to disable hostname validation available in the updated version of librdkafka I was able to connect from the container. Strangely, even with identical versions of openssl, librdkafka, and kafkacat I did not have any hostname validation issues running locally.</p>
<p>The openssl CLI has proven to be a very useful tool to troubleshoot TLS issues, so I inspected the endpoint as so:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl s_client <span class="nt">-connect</span> <brokername>
</code></pre></div></div>
<p>Finally, this revealed the issue. The output from the container showed an error that did not appear locally or from the brokers:</p>
<p><code class="language-plaintext highlighter-rouge">Verify return code: 68 (CA signature digest algorithm too weak)</code></p>
<p>Connections from the administrative container were finding that the SHA256 signed certificates were not sufficiently secure and were failing validation on connection. The certificates were created strictly following <a href="https://docs.confluent.io/current/security/security_tutorial.html#generating-keys-certs">Confluent’s instructions</a>, which uses the default signature strength from the keygen tool. And, of course, this worked perfectly fine for everything else. Luckily, the error message revealed by the openssl CLI lead me to a <a href="https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1638747.html">Debian mailing list archive</a> that finally solved the mystery. It turns out that in the newer versions of the Debian base image to which we had recently upgraded, the CipherString security level configured in <code class="language-plaintext highlighter-rouge">/etc/ssl/openssl.cnf</code> had been tightened. The solution presented was not feasible, as the configuration change would have had to be made by all clients throughout the company that used Kafka from Debian images. And we eventually needed to upgrade to SHA512, anyway. The only real option was to generate new CA certificates with the <code class="language-plaintext highlighter-rouge">-sha512</code> argument, as so:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl req <span class="nt">-new</span> <span class="nt">-x509</span> <span class="nt">-keyout</span> ca-key <span class="nt">-out</span> ca-cert <span class="nt">-days</span> 365 <span class="nt">-sha512</span>
</code></pre></div></div>
<p>For good measure, I also regenerated the broker certificates. Doing so required adding the <code class="language-plaintext highlighter-rouge">-sigalg SHA512WithRSA</code> option argument when running <code class="language-plaintext highlighter-rouge">keytool</code>, which looks something like this depending on your configuration:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>keytool <span class="nt">-genkey</span> <span class="nt">-keystore</span> <span class="k">${</span><span class="nv">NAME</span><span class="k">}</span> <span class="nt">-validity</span> 365 <span class="nt">-storepass</span> <span class="k">${</span><span class="nv">PASS</span><span class="k">}</span> <span class="nt">-keypass</span> <span class="k">${</span><span class="nv">PASS</span><span class="k">}</span> <span class="nt">-dname</span> <span class="s2">"CN=</span><span class="k">${</span><span class="nv">BROKER_NAME</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-keysize</span> 4096 <span class="nt">-keyalg</span> RSA <span class="nt">-deststoretype</span> pkcs12 <span class="nt">--ext</span> <span class="nv">$SAN</span> <span class="nt">-sigalg</span> SHA512WithRSA
</code></pre></div></div>
<p>After deploying the new certificates, all was corrected.</p>
<p>Although this was a major configuration change that must have created a lot of problems across the board, not just for Kafka clients, it’s hard to fault Debian for doing so given the focus on cryptographic strength at many organizations. I would, however, have thought to find a more official source of documentation on the issue than a mailing list archive. Hopefully this post will help deepen the options available to those googling the error message with hopes and prayers, as I had to do.</p>When I was suddenly unable to connect to the TLS listener with kafkacat, after a long period of troubleshooting I was able to track down the issue to changes in the default openssl configuration in Debian docker base images.TLS/SSL for Kafka in Docker Containers2019-12-31T00:00:00+00:002019-12-31T00:00:00+00:00https://jboulineau.github.io/blog/kafka/configure-tls-kafka-docker<p>Confluent provides <a href="https://docs.confluent.io/current/kafka/authentication_ssl.html">generally strong documentation</a> for configuring TLS/SSL. This post assumes you are familiar with this documentation, especially around key/certificate management. The documentation is for using the standard configuration files. For running in containers, the same settings are configured in environment variables, which are generally translatable nearly 1-1 with the keys in the files. More on that later. Confluent provides various <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> files to demonstrate how to translate the configuration values. <a href="https://github.com/confluentinc/cp-demo/blob/5.3.2-post/docker-compose.yml">This</a> is a good, fairly comprehensive, example from a platform demo. Additionally, <a href="https://github.com/confluentinc/cp-docker-images/tree/5.3.1-post/examples">a series of examples</a> are provided that cover a large number of scenarios.</p>
<p>However, after configuring TLS/SSL for Kafka using the Confluent Docker images, you may run into an error like this one when attempting to connect to the broker.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Failed authentication with /172.31.0.1<span class="o">(</span>SSL handshake failed<span class="o">)</span>
<span class="o">(</span>org.apache.kafka.common.network.Selector<span class="o">)</span>
</code></pre></div></div>
<p>After fiddling with <code class="language-plaintext highlighter-rouge">log4j</code> settings, you may uncover an error message like this one:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Inbound closed before receiving peer<span class="s1">'s close_notify: possible truncation attack?
</span></code></pre></div></div>
<p>Testing with <code class="language-plaintext highlighter-rouge">openssl s_client</code> may reveal further information:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CONNECTED<span class="o">(</span>00000003<span class="o">)</span>
140218707678080:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:
<span class="nt">---</span>
no peer certificate available
<span class="nt">---</span>
No client certificate CA names sent
<span class="nt">---</span>
SSL handshake has <span class="nb">read </span>5 bytes and written 407 bytes
Verification: OK
<span class="nt">---</span>
New, <span class="o">(</span>NONE<span class="o">)</span>, Cipher is <span class="o">(</span>NONE<span class="o">)</span>
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify <span class="k">return </span>code: 0 <span class="o">(</span>ok<span class="o">)</span>
<span class="nt">---</span>
</code></pre></div></div>
<p>If you’re here, you are no doubt as frustrated as I was. I was <em>certain</em> that I had the correct certificates and configurations in place. It turns out, there is a hidden twist to the configuration. To understand what was going on, a deeper at the container configurations is required.</p>
<p>For configuring TLS/SSL in Confluent containers, there are two methods that may be used.</p>
<h3 id="example-1">Example 1</h3>
<p>The first is to set the path to the keystore files and include the passwords in plain text.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">KAFKA_SSL_KEYSTORE_LOCATION</span><span class="o">=</span><path>
<span class="nv">KAFKA_SSL_KEYSTORE_PASSWORD</span><span class="o">=</span><plain-text-password>
<span class="nv">KAFKA_SSL_TRUSTSTORE_LOCATION</span><span class="o">=</span><path>
<span class="nv">KAFKA_SSL_TRUSTSTORE_PASSWORD</span><span class="o">=</span><plain-text-password>
</code></pre></div></div>
<p>Each <code class="language-plaintext highlighter-rouge">LOCATION</code> variable is the full path to the keystore file wherever you decide to mount them.</p>
<h3 id="example-2">Example 2</h3>
<p>The example <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> files prefer the method of setting keystore filenames and using credential files to store the passwords for the keystores. This is clearly preferable for production as secrets files can be injected at runtime as part of your CI/CD pipeline and you can keep sensitive values out of source control. This method assumes you have mounted the necessary files into the <code class="language-plaintext highlighter-rouge">/etc/kafka/secrets</code> path of the container.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>KAFKA_SSL_KEYSTORE_FILENAME: <filename>
KAFKA_SSL_KEYSTORE_CREDENTIALS: <filename>
KAFKA_SSL_KEY_CREDENTIALS: <filename>
KAFKA_SSL_TRUSTSTORE_FILENAME: <filename>
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: <filename>
</code></pre></div></div>
<p>What I found is that when using the first option the TLS endpoint worked as expected, but the second method resulted in the errors above. This demonstrated, at least, that the keystores and certificates were created properly. But, why wouldn’t the second (much preferred) configuration option work?</p>
<p>The answer is to be found in the <a href="https://github.com/confluentinc/cp-docker-images/blob/5.3.1-post/debian/kafka/include/etc/confluent/docker/configure#L65">configure script</a> for the Confluent Kafka Docker image, which is executed by the <a href="https://github.com/confluentinc/cp-docker-images/blob/5.3.1-post/debian/kafka/include/etc/confluent/docker/run">entry point script</a>. Line 65 of the script looks at the <code class="language-plaintext highlighter-rouge">KAFKA_ADVERTIZED_LISTENERS</code> environment variable to determine whether or not SSL is configured.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">[[</span> <span class="nv">$KAFKA_ADVERTISED_LISTENERS</span> <span class="o">==</span> <span class="k">*</span><span class="s2">"SSL://"</span><span class="k">*</span> <span class="o">]]</span>
</code></pre></div></div>
<p>The script requires that the name of the TLS listener <strong>must</strong> have <code class="language-plaintext highlighter-rouge">SSL</code> as the final three characters. In my case, I was using <code class="language-plaintext highlighter-rouge">SSL_INTERNAL</code> as the name of my listener, which did not match the pattern. Changing the name to <code class="language-plaintext highlighter-rouge">INTERNAL_SSL</code> resolved the problem.</p>
<p>The code section that runs in the conditional translates the environment variables set in <strong>example 2</strong> into the environment variables of <strong>example 1</strong>, which are the ones actually read by Kafka during startup. This means that if you use the method of <strong>example 1</strong>, you can use whatever listener name you want.</p>
<p>There is now an open (internal) ticket with Confluent to update the documentation to make this behavior clear. Perhaps they will also reconsider the implementation of the image so that it is less fragile.</p>
<p><em>Thanks to Ryan Alexander of Confluent for his help.</em></p>While configuring TLS/SSL for Confluent Kafka is straightforward, there are twists when running in Docker containers. This posts covers what I discovered that isn't (as of time of writing) covered in the official documentation.Dealing With Bad Records in Kafka2019-06-13T00:00:00+00:002019-06-13T00:00:00+00:00https://jboulineau.github.io/blog/kafka/dealing-with-bad-records-in-kafka<p>There is currently a rather serious flaw in the Java <code class="language-plaintext highlighter-rouge">KafkaConsumer</code> when combined with <code class="language-plaintext highlighter-rouge">KafkaAvroDeserializer</code>, which is <a href="https://docs.confluent.io/current/schema-registry/schema_registry_tutorial.html">used to deserialize records</a> when their schemas are stored in Schema Registry. A <a href="https://issues.apache.org/jira/browse/KAFKA-4740">critical issue</a> has been opened, but it hasn’t been updated since December 2018.</p>
<p>In brief, the issue is that when a record is encountered that cannot be deserialized from Avro (a.k.a a poison pill) an exception will be thrown:</p>
<blockquote>
<p>org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition topic-0 at offset 2
If needed, please seek past the record to continue consumption.</p>
</blockquote>
<p>This is, of course, how it should behave. However, it fails in an <strong>unrecoverable</strong> manner. Because it cannot be deserialized the record is not added to the collection returned by <code class="language-plaintext highlighter-rouge">poll()</code>, which means running a <code class="language-plaintext highlighter-rouge">commit()</code> variant won’t do anything. And, despite the instructions in the exception meessage, you can’t easily <code class="language-plaintext highlighter-rouge">seek()</code> past the record because the necessary partition and offset information would be in the <code class="language-plaintext highlighter-rouge">ConsumerRecord</code> instance, which is never instantiated. This results in a loop of fail, as <code class="language-plaintext highlighter-rouge">poll()</code> will continue returning the same record repeatedly with no provided way to get past the record.</p>
<p>Needless to say, this is disappointing for such a widely used and essential platform for just about anyone doing data engineering or event driven architecture of any sort these days. Hopefully the <a href="https://issues.apache.org/jira/browse/KAFKA-4740">issue</a> will be addressed, although it’s going to be tricky to do without introducing breaking semantic changes. The alternative presented in the issue of passing partition id and offset through the exception object isn’t exactly clean, but may be the best option. Regardless, in the meantime there are options to explore. And here they are!</p>
<h2 id="correct-the-deserializer">‘Correct’ the deserializer</h2>
<p>Since the heart of the problem is that the <code class="language-plaintext highlighter-rouge">KafkaConsumer</code> doesn’t handle <code class="language-plaintext highlighter-rouge">SerializationException</code> in a recoverable manner, one solution would be simply not to throw <code class="language-plaintext highlighter-rouge">SerializationException</code> anymore. By extending <code class="language-plaintext highlighter-rouge">io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer</code> (a la <code class="language-plaintext highlighter-rouge">KafkaAvroDeserializer</code>) we can make this happen.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">Object</span> <span class="nf">deserialize</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">,</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">bytes</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">deserialize</span><span class="o">(</span><span class="n">bytes</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">SerializationException</span> <span class="n">e</span><span class="o">)</span>
<span class="o">{</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">Object</span> <span class="nf">deserialize</span><span class="o">(</span><span class="nc">String</span> <span class="n">s</span><span class="o">,</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">bytes</span><span class="o">,</span> <span class="nc">Schema</span> <span class="n">readerSchema</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">deserialize</span><span class="o">(</span><span class="n">bytes</span><span class="o">,</span> <span class="n">readerSchema</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">SerializationException</span> <span class="n">e</span><span class="o">)</span>
<span class="o">{</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now, inject the class into <code class="language-plaintext highlighter-rouge">ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG</code> …</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">VALUE_DESERIALIZER_CLASS_CONFIG</span><span class="o">,</span> <span class="o"><</span><span class="n">my_deserializer_name</span><span class="o">>.</span><span class="na">class</span><span class="o">);</span>
</code></pre></div></div>
<p>… and catch those null objects.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">ConsumerRecords</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Payment</span><span class="o">></span> <span class="n">records</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="na">poll</span><span class="o">(</span><span class="nc">Duration</span><span class="o">.</span><span class="na">ofMillis</span><span class="o">(</span><span class="mi">100</span><span class="o">));</span>
<span class="k">for</span> <span class="o">(</span><span class="kd">final</span> <span class="nc">ConsumerRecord</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Payment</span><span class="o">></span> <span class="n">record</span> <span class="o">:</span> <span class="n">records</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">key</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="na">key</span><span class="o">();</span>
<span class="k">if</span> <span class="o">(</span><span class="n">record</span><span class="o">.</span><span class="na">value</span><span class="o">()</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"A poison pill record was encountered."</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">Payment</span> <span class="n">value</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">printf</span><span class="o">(</span><span class="s">"key = %s, value = %s%n"</span><span class="o">,</span> <span class="n">key</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">ClassCastException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"This record is not a `Payment` event."</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">consumer</span><span class="o">.</span><span class="na">commitSync</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>
<p>It’s not great. Converting exceptions into null objects hurts, but, given the alternatives below, it may be the best option. It’s the one I’m using.</p>
<h2 id="delete-the-record-from-the-topic">Delete the record from the topic</h2>
<p>It is possible to delete the record from the topic manually by using the <code class="language-plaintext highlighter-rouge">kafka-delete-records.sh</code>. I haven’t tried this (yet), so YMMV, use at your own use, etc. <a href="http://www.alternatestack.com/development/kafka-tools-kafka-delete-records/">Here</a> is a blog post on the subject.</p>
<p>Of course, relying on this option means that anytime your consumer(s) encounter(s) a poison pill you’re down until your Kafka administrators can fix the issue. And <strong>Bad Things</strong> can always happen when you’re munging from the command line.</p>
<h2 id="use-the-streams-api">Use the Streams API</h2>
<p>(Not tested) Interestingly, the problem has been fixed in the Streams API, essentially by the same method above of implementing the ‘catch exception and move on’ method. There was an <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers">improvement proposal</a> on the subject that became a <a href="https://issues.apache.org/jira/browse/KAFKA-5157">JIRA issue</a> and the fix was <a href="https://github.com/apache/kafka/pull/3423">merged</a>.</p>
<p>The downside here is that the Streams API is meant for the very specific usage patterns of stream processing. It may or may not fit your use case.</p>
<h2 id="use-spring">Use Spring</h2>
<p>(Not tested) For those using the Spring framework, there is a <a href="https://docs.spring.io/spring-kafka/docs/2.2.0.RELEASE/reference/html/_reference.html#error-handling-deserializer">fix in version 2.2</a>. Here’s a <a href="https://www.confluent.io/blog/spring-for-apache-kafka-deep-dive-part-1-error-handling-message-conversion-transaction-support">Confluent post</a> to help.</p>
<p>The downside of this option is Spring.</p>
<h2 id="use-a-libdrkafka-client">Use a libdrkafka client</h2>
<p>(Not tested) Colleagues assure me that <code class="language-plaintext highlighter-rouge">libdrkafka</code> based clients, such as in <a href="https://docs.confluent.io/current/clients/confluent-kafka-python/index.html#consumer">Python</a>, don’t have this issue. If I get the chance I’ll test this one out and update this post.</p>
<h2 id="parse-the-error-text-and-seek-past-the-bad-record">Parse the error text and seek past the bad record</h2>
<p>We’re well into hack territory here, but it might get you through in a pinch. Let’s not dwell. There’s some code below, if you must.</p>
<p>Just don’t tell anyone I helped you do this.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.confluent.kafka.serializers.subject.TopicRecordNameStrategy</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.clients.consumer.ConsumerConfig</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.clients.consumer.ConsumerRecord</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.clients.consumer.ConsumerRecords</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.clients.consumer.KafkaConsumer</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.confluent.kafka.serializers.KafkaAvroDeserializer</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">io.confluent.kafka.serializers.KafkaAvroDeserializerConfig</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.common.TopicPartition</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.common.errors.SerializationException</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.common.serialization.StringDeserializer</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.time.Duration</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.regex.Pattern</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.regex.Matcher</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.Collections</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">java.util.Properties</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">SampleConsumer</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">TOPIC</span> <span class="o">=</span> <span class="s">"payment"</span><span class="o">;</span>
<span class="nd">@SuppressWarnings</span><span class="o">(</span><span class="s">"InfiniteLoopStatement"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">Consume</span><span class="o">()</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">Properties</span> <span class="n">props</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Properties</span><span class="o">();</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">ENABLE_AUTO_COMMIT_CONFIG</span><span class="o">,</span> <span class="s">"false"</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">BOOTSTRAP_SERVERS_CONFIG</span><span class="o">,</span> <span class="s">"localhost:9092"</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">GROUP_ID_CONFIG</span><span class="o">,</span> <span class="s">"my-consumer"</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">AUTO_OFFSET_RESET_CONFIG</span><span class="o">,</span> <span class="s">"earliest"</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">KEY_DESERIALIZER_CLASS_CONFIG</span><span class="o">,</span> <span class="nc">StringDeserializer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">VALUE_DESERIALIZER_CLASS_CONFIG</span><span class="o">,</span> <span class="nc">KafkaAvroDeserializer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="na">MAX_POLL_INTERVAL_MS_CONFIG</span><span class="o">,</span> <span class="mi">100000</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">AbstractKafkaAvroSerDeConfig</span><span class="o">.</span><span class="na">SCHEMA_REGISTRY_URL_CONFIG</span><span class="o">,</span> <span class="s">"http://localhost:8081"</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">KafkaAvroDeserializerConfig</span><span class="o">.</span><span class="na">SPECIFIC_AVRO_READER_CONFIG</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">KafkaAvroDeserializerConfig</span><span class="o">.</span><span class="na">KEY_SUBJECT_NAME_STRATEGY</span><span class="o">,</span> <span class="nc">TopicRecordNameStrategy</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getName</span><span class="o">());</span>
<span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">KafkaAvroDeserializerConfig</span><span class="o">.</span><span class="na">VALUE_SUBJECT_NAME_STRATEGY</span><span class="o">,</span> <span class="nc">TopicRecordNameStrategy</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getName</span><span class="o">());</span>
<span class="cm">/* Sadly, we need to do some string parsing to deal with 'poison pill' records (i.e. any message that cannot be
de-serialized by KafkaAvroDeserializer, most likely because they weren't produced using Schema Registry) so we
need to set up some regex things
*/</span>
<span class="kd">final</span> <span class="nc">Pattern</span> <span class="n">offsetPattern</span> <span class="o">=</span> <span class="nc">Pattern</span><span class="o">.</span><span class="na">compile</span><span class="o">(</span><span class="s">"\\w*offset*\\w[ ]\\d+"</span><span class="o">);</span>
<span class="kd">final</span> <span class="nc">Pattern</span> <span class="n">partitionPattern</span> <span class="o">=</span> <span class="nc">Pattern</span><span class="o">.</span><span class="na">compile</span><span class="o">(</span><span class="s">"\\w*"</span> <span class="o">+</span> <span class="no">TOPIC</span> <span class="o">+</span> <span class="s">"*\\w[-]\\d+"</span><span class="o">);</span>
<span class="nc">KafkaConsumer</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">LoanCreated</span><span class="o">></span> <span class="n">consumer</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">KafkaConsumer</span><span class="o"><>(</span><span class="n">props</span><span class="o">);</span>
<span class="n">consumer</span><span class="o">.</span><span class="na">subscribe</span><span class="o">(</span><span class="nc">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">(</span><span class="no">TOPIC</span><span class="o">));</span>
<span class="c1">// Consume messages</span>
<span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">ConsumerRecords</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">LoanCreated</span><span class="o">></span> <span class="n">records</span> <span class="o">=</span> <span class="n">consumer</span><span class="o">.</span><span class="na">poll</span><span class="o">(</span><span class="nc">Duration</span><span class="o">.</span><span class="na">ofMillis</span><span class="o">(</span><span class="mi">1</span><span class="o">));</span>
<span class="k">for</span> <span class="o">(</span><span class="kd">final</span> <span class="nc">ConsumerRecord</span><span class="o"><</span><span class="nc">String</span><span class="o">,</span> <span class="nc">LoanCreated</span><span class="o">></span> <span class="n">record</span> <span class="o">:</span> <span class="n">records</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">key</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="na">key</span><span class="o">();</span>
<span class="k">try</span> <span class="o">{</span>
<span class="cm">/* A record can be successfully de-serialized, but is not coercable into the type we need. In
the case of this example, we're looking for LoanCreated records, but we are also producing
Payment records. */</span>
<span class="kd">final</span> <span class="nc">LoanCreated</span> <span class="n">value</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">printf</span><span class="o">(</span><span class="s">"key = %s, value = %s%n"</span><span class="o">,</span> <span class="n">key</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span>
<span class="c1">// do work here</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">ClassCastException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Record is not the specified type ... skipping"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">consumer</span><span class="o">.</span><span class="na">commitSync</span><span class="o">();</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">SerializationException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">text</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="na">getMessage</span><span class="o">();</span>
<span class="c1">// Parse the error message to get the partition number and offset, in order to `seek` past the poison pill.</span>
<span class="nc">Matcher</span> <span class="n">mPart</span> <span class="o">=</span> <span class="n">partitionPattern</span><span class="o">.</span><span class="na">matcher</span><span class="o">(</span><span class="n">text</span><span class="o">);</span>
<span class="nc">Matcher</span> <span class="n">mOff</span> <span class="o">=</span> <span class="n">offsetPattern</span><span class="o">.</span><span class="na">matcher</span><span class="o">(</span><span class="n">text</span><span class="o">);</span>
<span class="n">mPart</span><span class="o">.</span><span class="na">find</span><span class="o">();</span>
<span class="nc">Integer</span> <span class="n">partition</span> <span class="o">=</span> <span class="nc">Integer</span><span class="o">.</span><span class="na">parseInt</span><span class="o">(</span><span class="n">mPart</span><span class="o">.</span><span class="na">group</span><span class="o">().</span><span class="na">replace</span><span class="o">(</span><span class="no">TOPIC</span> <span class="o">+</span> <span class="s">"-"</span><span class="o">,</span> <span class="s">""</span><span class="o">));</span>
<span class="n">mOff</span><span class="o">.</span><span class="na">find</span><span class="o">();</span>
<span class="nc">Long</span> <span class="n">offset</span> <span class="o">=</span> <span class="nc">Long</span><span class="o">.</span><span class="na">parseLong</span><span class="o">(</span><span class="n">mOff</span><span class="o">.</span><span class="na">group</span><span class="o">().</span><span class="na">replace</span><span class="o">(</span><span class="s">"offset "</span><span class="o">,</span> <span class="s">""</span><span class="o">));</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="nc">String</span><span class="o">.</span><span class="na">format</span><span class="o">(</span>
<span class="s">"'Poison pill' found at partition {0}, offset {1} .. skipping"</span><span class="o">,</span> <span class="n">partition</span><span class="o">,</span> <span class="n">offset</span><span class="o">));</span>
<span class="n">consumer</span><span class="o">.</span><span class="na">seek</span><span class="o">(</span><span class="k">new</span> <span class="nc">TopicPartition</span><span class="o">(</span><span class="no">TOPIC</span><span class="o">,</span> <span class="n">partition</span><span class="o">),</span> <span class="n">offset</span> <span class="o">+</span> <span class="mi">1</span><span class="o">);</span>
<span class="c1">// Continue on</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>A single bad record (a.k.a poison pill) on a Kafka topic can ruin your day. KafkaConsumer does not deal with these records gracefully. Here I cover strategies on how to address this issue.Automating SQL Server Databases with Docker2019-05-13T00:00:00+00:002019-05-13T00:00:00+00:00https://jboulineau.github.io/blog/sql-server/sql-server-database-in-docker<p>Running SQL Server in a Docker container has been possible for quite a while now. Very shortly after it was announced that SQL Server 2017 could run on Linux, Docker images were <a href="https://blog.docker.com/2017/09/microsoft-sql-on-docker-ee/">available</a> on the <a href="https://hub.docker.com/_/microsoft-mssql-server">Docker repository</a>. The speed and convenience of being able to have a running SQL Server instance in seconds is a huge productivity saver. Installing and managing SQL Server on a development workstation has never been an attractive prospect. It is time consuming, installs <em>tons</em> of dependencies that are not removed even if you uninstall SQL Server, and can drain resources even when it’s not in use. Managing the database is often a drag on contemporary development practices.</p>
<p>But, having SQL Server in Docker isn’t really enough. Even after having the instance available, it is still often an onerous process to deploy the database. What I need is the ability to spin up one or more <strong>databases</strong> with the same ease as deploying an instance of my service. ‘Dockerizing’ my database projects has had an immediate impact on my work efficiency due to the ease in being able to create on demand database instances with the most recent version of the code, unattended, in a matter of minutes. Best of all, the image can be automagically registered in my company’s Docker Trusted Repository (DTR) so that anyone can include the image in, say, <code class="language-plaintext highlighter-rouge">docker-compose</code> for their own purposes.</p>
<p>Fortunately, by leveraging the capabilities of the <a href="https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/data-tier-applications?view=sql-server-2017">Data-Tier Application</a> and related DacFx framework it is possible to include one or more databases in the SQL Server container. The path to the Data-Tier Application is through <a href="https://docs.microsoft.com/en-us/sql/ssdt/download-sql-server-data-tools-ssdt?view=sql-server-2017">SQL Server Data Tools</a> within Visual Studio. A tutorial on using SSDT for SQL Server database projects is beyond the scope of this post. However, <a href="https://www.mssqltips.com/sqlservertutorial/9000/overview-of-ssdt-sql-project-tutorial/">here</a>, <a href="https://docs.microsoft.com/en-us/previous-versions/sql/sql-server-data-tools/hh272702(v=vs.103)">here</a>, or <a href="https://sqlbits.com/Sessions/Event11/Real_world_SSDT">here</a> are some places to start.</p>
<p>Once you have your database project created (which is possible using an <a href="https://www.mssqltips.com/sqlservertip/2971/creating-a-visual-studio-database-project-for-an-existing-sql-server-database/">existing database</a>), you’re ready to start.</p>
<p>Now, let’s create a <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile</a> in the project folder. Start with the Microsoft base image.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> mcr.microsoft.com/mssql/server:2017-latest</span>
</code></pre></div></div>
<p>The default product for the image is ‘Developer,’ the free single-user edition of SQL Server Enterprise. If used properly there is no cost for this edition, but make sure you understand the license so you stay in compliance. Next, set the required environment variables for SQL Server.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">ENV</span><span class="s"> ACCEPT_EULA=Y</span>
<span class="k">ENV</span><span class="s"> SA_PASSWORD=<strong_password></span>
</code></pre></div></div>
<p>Yes, the SA password is set in plain text in the Dockerfile, which is typically committed to version control. For development activities this is probably fine (just don’t use a <strong>REAL</strong> password), but if you are intending on utilizing the image as part of a production deployment of your service, it is most certainly not. It is possible to externalize the secret to an <a href="https://vsupalov.com/docker-arg-env-variable-guide/">environment file</a> or pass the value through <code class="language-plaintext highlighter-rouge">docker build</code> using <code class="language-plaintext highlighter-rouge">ARG</code> in such a scenario.</p>
<p>Next, the <code class="language-plaintext highlighter-rouge">COPY</code> command here is copying everything in the same directory as the Dockerfile into /var/opt/ within the container.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">COPY</span><span class="s"> . /var/opt/<project_name></span>
</code></pre></div></div>
<p>A digression is in order. Although at this point the entire database project exists within the container, to date Microsoft has not added support for database projects in dotnet core. This means it is not possible to generate the deployment artifact (.dacpac) within the Linux container. It is especially frustrating since they <strong>have</strong> ported the CLI for .dacpac deployments to Linux, so you can deploy one, you just can’t create one. At time of writing, there doesn’t seem to be an open issue for the <a href="https://github.com/microsoft/msbuild/issues?utf8=%E2%9C%93&q=database">MSBuild project</a>, either. Therefore, it is necessary to ensure that a build is run for the project either through MSBuild or Visual Studio <strong>before</strong> the build for the image, an annoying extra step that can’t be automated in the Dockerfile. In projects I manage this is currently addressed by prohibiting PRs that do not include a .dacpac for code changes. At some point I’ll try to figure out how to enforce that further with a <a href="https://pre-commit.com/">pre-commit</a> hook. Let’s get back to the Dockerfile.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">RUN </span>apt-get update <span class="se">\
</span> <span class="o">&&</span> apt-get <span class="nb">install </span>unzip dotnet-sdk-2.2 <span class="nt">-y</span> <span class="nt">--no-install-recommends</span> <span class="se">\
</span> <span class="o">&&</span> wget <span class="nt">-q</span> <span class="nt">-O</span> /var/opt/sqlpackage.zip https://go.microsoft.com/fwlink/?linkid<span class="o">=</span>2069122 <span class="se">\
</span> <span class="o">&&</span> unzip <span class="nt">-qq</span> /var/opt/sqlpackage.zip <span class="nt">-d</span> /var/opt/sqlpackage <span class="se">\
</span> <span class="o">&&</span> <span class="nb">rm</span> /var/opt/sqlpackage.zip <span class="se">\
</span> <span class="o">&&</span> <span class="nb">chmod</span> +x /var/opt/sqlpackage/sqlpackage <span class="se">\
</span></code></pre></div></div>
<p>Microsoft’s base image for SQL Server is missing software we need. If you are going to run a unit test suite, the dotnet core sdk is required. If you’re not unit testing, perhaps because all of your DML is in your app, you can probably remove dotnet-sdk-2.2 from the install. You will, however, need to download <a href="https://docs.microsoft.com/en-us/sql/tools/sqlpackage?view=sql-server-2017">sqlpackage</a> (the DacFX CLI). In order to extract from the .zip, unzip is required.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code> && mv /var/opt/<project_dir>/bin/Debug/master.dacpac /var/opt/<project_dir>/bin/Debug/MASTER.DACPAC \
&& mv /var/opt/<project_dir>/bin/Debug/msdb.dacpac /var/opt/<project_dir>/bin/Debug/MSDB.DACPAC \
</code></pre></div></div>
<p>Often T-SQL objects will include references to objects stored in system databases. These dependencies can be added to the project in the form of their respective .dacpac, which allows Visual Studio to resolve the references. In a discovery that wasted a significant chunk of time, sqlpackage only looks for dependencies in upper case names <code class="language-plaintext highlighter-rouge">¯\_(ツ)_/¯</code>. In Windows, this doesn’t matter, but in Linux it certainly does. It looks like a missed test case in the dotnet core port. Let’s move on…</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code> && (/opt/mssql/bin/sqlservr --accept-eula & ) | grep -q "Service Broker manager has started" \
&& /var/opt/sqlpackage/sqlpackage /a:Publish /tdn:<db_name> \
/pr:/var/opt/<project_dir>/docker.publish.xml /sf:/var/opt/<project_dir>/bin/Debug/<project-name>.dacpac \
/p:IncludeCompositeObjects=true /tu:sa /tp:$SA_PASSWORD \
&& (/opt/mssql-tools/bin/sqlcmd -S localhost -d <db_name> -U SA -P $SA_PASSWORD -Q 'ALTER DATABASE <db_name> SET RECOVERY SIMPLE') \
</code></pre></div></div>
<p>Here we start SQL Server and use sqlpackage to deploy the database. You’ll notice that a publish profile has been created and stored with the project specifically for deploying to SQL Server running in the container. It’s also a good idea to make sure the database is set to <code class="language-plaintext highlighter-rouge">SIMPLE</code> recovery for testing purposes to keep the log file small. You probably don’t want to do this in prod. Note also that this Dockerfile doesn’t establish external volumes for the data and log files, which you will probably want to do if using this in a production environment.</p>
<p>This is also a good place to insert any other SQL commands that makes sense for you using <code class="language-plaintext highlighter-rouge">sqlcmd</code>. For instance, this is where I also include the creation of any service accounts that need to have access to my database for whatever it is I’m testing. The key is to strive for zero administration or setup required whenever the container is run.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code> && cd /var/opt/<project_dir>/<test_dir>/ && dotnet test \
</code></pre></div></div>
<p>In most database projects I have a testing suite. MS Test, xUnit, and NUnit are available in dotnet core. Visual Studio works very nicely with MS Test, which includes a SQL specific test harness that can be handy, if somewhat restricted. This line in the Dockerfile executes the tests in the project.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code> && pkill sqlservr \
&& apt-get remove dotnet-sdk-2.2 -y \
&& apt-get autoremove -y \
&& rm -rf /usr/share/dotnet \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /var/opt/<project_dir> \
&& rm -rf /var/opt/sqlpackage
</code></pre></div></div>
<p>Finally, the last steps shutdown SQL Server and remove the software that was installed. This helps to keep an already large image as small as possible. The end result, depending on what is involved in deploying your database and running your tests, should be an image which is 1.6 - 2 GB in size.</p>
<h2 id="conclusion">Conclusion</h2>
<p>For future enhancement I intend on implementing test data population as part of this process, so that it will be possible to spin up databases very quickly for integration testing purposes. Restoring a database from backup should be quite simple to automate, assuming the backups are stored on a network-accessible path, so that is one option I am considering. That would also be a great option to rapidly deploy databases for shared environments, though that would call for the additional enhancement of being able to run masking/anonymization operations on the restored data set. But, like you have seen, it is possible to automate just about anything in the build of a Docker image.</p>Installing and managing SQL Server on a development workstation has never been an attractive prospect. It is time consuming, installs _tons_ of dependencies that are not removed even if you uninstall SQL Server, and can drain resources even when it's not in use. Managing the database is often a drag on contemporary development practices. Automating database deployments in a Docker container can be a big boost in development efficiency.CS0579 Duplicate Attribute Error with .NET Core2018-06-08T00:00:00+00:002018-06-08T00:00:00+00:00https://jboulineau.github.io/blog/dotnet/core-CS0579-error<p>UPDATED: 2020-12-23</p>
<p>I’ll leave the below for posterity, but recently I came across the real reason for the error. In this <a href="https://github.com/dotnet/core/issues/4837">issue</a> it was pointed out that your .net core solutions have to be structured in a particular way. Specifically, make sure that you do not create projects in the same directory as your main project.</p>
<hr />
<h2 id="problem-description">Problem description</h2>
<p>I recently started working on my first .NET Core project and quickly ran into a problem. After adding a class library project, the second time I ran <code class="language-plaintext highlighter-rouge">dotnet run</code> I received the following errors:</p>
<blockquote>
<p>error CS0579: Duplicate ‘System.Reflection.AssemblyCompanyAttribute’ attribute
error CS0579: Duplicate ‘System.Reflection.AssemblyConfigurationAttribute’ attribute
error CS0579: Duplicate ‘System.Reflection.AssemblyFileVersionAttribute’ attribute
error CS0579: Duplicate ‘System.Reflection.AssemblyInformationalVersionAttribute’ attribute
error CS0579: Duplicate ‘System.Reflection.AssemblyProductAttribute’ attribute
error CS0579: Duplicate ‘System.Reflection.AssemblyTitleAttribute’ attribute
error CS0579: Duplicate ‘System.Reflection.AssemblyVersionAttribute’ attribute</p>
</blockquote>
<h2 id="tldr-solution">tl;dr Solution</h2>
<p>My Google-fu revealed the solution buried in a <a href="https://github.com/dotnet/cli/issues/4783">GitHub issue thread</a>. Add the following line to your <code class="language-plaintext highlighter-rouge">*.csproj</code> files:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><GenerateAssemblyInfo></span>false<span class="nt"></GenerateAssemblyInfo></span>
</code></pre></div></div>
<p>As the name indicates, this will prevent the build process from generating the AssemblyInfo.cs file in project <code class="language-plaintext highlighter-rouge">obj</code> directories, thus preventing the duplicate attribute conflict.</p>
<h2 id="details">Details</h2>
<p><strong>Disclaimer</strong>: I am far from a .NET expert so there is probably more to this story.</p>
<p>The error is a result of the build process generating <code class="language-plaintext highlighter-rouge">AssemblyInfo.cs</code> files in each project <code class="language-plaintext highlighter-rouge">obj</code> directory. This file exists to provide MSBuild metadata about the resulting assembly that can be attached to the build artifact and utilized by anyone who wants such information. You can see your assembly info in Windows by selecting your build artifact, right click -> properties. The following shows the results with GenerateAssemblyInfo set to false. As you can see, the metadata is set to default values.</p>
<p><img src="/assets/images/assemblyinfo_false.PNG" /></p>
<p>It’s not clear why the default <code class="language-plaintext highlighter-rouge">.csproj</code> file generated by <code class="language-plaintext highlighter-rouge">dotnet new</code> results in this behavior. The closest I came across was a mention of the philosophy to keep the file as terse as possible. Of course, I’m completely in support of this as <code class="language-plaintext highlighter-rouge">.csproj</code> tends to be a dense mess in Visual Studio. However, this particular scenario puts the onus on the developer to edit the file to eliminate an error which is almost guaranteed to occur in a project of any complexity. A better design decision would have probably been to default to not generating <code class="language-plaintext highlighter-rouge">AssemblyInfo.cs</code>.</p>I recently started working with .NET Core and almost immediately came across an CS0579 Duplicate Attribute Error. This is what I discovered and how I fixed the issue.Kappa Architecture2018-05-29T00:00:00+00:002018-05-29T00:00:00+00:00https://jboulineau.github.io/blog/architecture/kappa-architecture<p>The previous post in this series covered the Lambda Architecture, which conceives of a solution with three layers: the batch layer, the speed layer, and the serving layer. Implemented successfully many times, the Lambda Architecture provides a complete solution that ensures (eventual) accuracy of processing as well as real-time processing results. However, this achievement does not come without costs.</p>
<p>Jay Kreps outlined these costs in an influential blog post. In <a href="http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html">Questioning the Lambda Architecture</a>, Kreps agrees with the basic propositions of the Lambda architecture, namely:</p>
<ol>
<li>Data consistency and accuracy are problems inherent in real-time distributed systems.</li>
<li>Batch processing, in the context of modern architectural patterns and techniques, is an effective way of overcoming the limitations of real-time distributed systems.</li>
</ol>
<p>However, Kreps focused on the Lambda Architecture’s weakness: complexity. While it is great, Kreps argues, that the Lambda Architecture can maintain conceptual consistency through implementing the same algorithms across its layers, the fact is that they are incongruent implementations. Each processing layer requires it’s own set of technologies and code bases that must be supported and maintained independently. And the serving layer is tasked with the potentially difficult challenge of reconciling results that may be overlapping or otherwise divergent. While clearly achievable, the Lambda Architecture is not as simple as it could be.</p>
<p>The problem, Kreps proposed, is not so much conceptual as technical. The Lambda principle that a data system is simply an assortment of functions applied to data is fundamentally correct, useful, and illuminating. However, the technological challenges of applying functions to small windows of data (real-time streams) are different than the problems of applying the same functions to all data at once. For real-time streams the primary concern is processing latency and is, therefore, typically network and CPU bound. For batch the primary concern is throughput, trying to optimize for the sheer volume of data being processed. It is typically disk I/O bound. Perhaps the reason why Marz had two processing layers in the Lambda Architecture is not because it is inherently required, but because no technological capability existed circa 2011 that serve both the requirements of real-time layer and the batch layer. For Kreps, the key remains his observation that the <a href="https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying">unifying data abstraction</a> is the immutable transaction log. Having a persistent, ordered, log of all events that occur in your environment enables both real-time and batch processes.</p>
<p>Therefore, central to the architecture is Kreps’ most famous project, <a href="https://kafka.apache.org">Apache Kafka</a>. Kafka, he argued, checks all of the boxes required for the Lambda Architecture. By providing low-latency, distributed, event topics it can allow rapid access to events as they occur for real-time processing in a pub/sub pattern. By persisting these events in an ordered immutable log data structure that can be replayed at high-throughput, it can also serve batch needs. Following the Lambda principle, whereas a real-time algorithm is one that is applied to a ‘window’ of a single event, a batch algorithm is one that is applied to the entire log of events in total. In Kafka, this simply means you can implement batch processing simply by executing the same real-time code starting from the beginning of the log.</p>
<p>Kreps proposed a modification that looks something like this:</p>
<p><center><img src="/assets/images/kappa_simple.jpg" alt="Kappa Architecture" /></center></p>
<p><a href="https://www.linkedin.com/pulse/from-lambda-architecture-kappa-using-apache-beam-siddharth-mittal/">Image Source</a></p>
<p>By adding Kafka as the central point of the architecture, the three conceptual layers of the Lambda Architecture can be merged into two. Inserting the immutable transaction log also serves the broader concerns of application integration. Readers of this series will observe that this is similar to the solution proposed in the <a href="/blog/architecture/improving-the-real-time-app">5th part of this series</a>.</p>
<p>Note that the distributed immutable event log data structure and related semantics are gaining traction. Kafka is widely recognized as a critical component of many modern solutions, particularly in a micro-service architecture. Microsoft recently jumped into the fray adding a <a href="https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview">Kafka API</a> to its Azure Event Hubs service. <a href="https://aws.amazon.com/kinesis/">AWS Kinesis</a> has enabled similar capabilities since late 2013.</p>
<p>Near the end of his post Kreps quibbled over the idea of naming his architecture. In my view he was right to do so as the Kappa architecture validates the fundamental concept of the Lambda Architecture. Pointing out that the same approach can be achieved with less complexity by utilizing a new type of platform may not justify a new name. Perhaps it would be better to have suggested Lambda 2.0. Regardless, it remains in the lexicon of distributed data systems.</p>The 'Intro to Data Streaming' series continues with an overview of the Kappa Architecture, a proposed enhancement to the Lambda Architecture. While agreeing with the basic formula it proposes eliminating some of the technical overhead and complexity.Running Hadoop on WSL2018-05-23T00:00:00+00:002018-05-23T00:00:00+00:00https://jboulineau.github.io/blog/hadoop/hadoop-on-wsl<p>I’ve been using Linux a lot lately, but it’s become annoying switching back-and-forth between Windows and the Linux VM. I’ve been thinking about switching to Linux as my primary OS, which I may still do, but I thought I’d give <a href="https://docs.microsoft.com/en-us/windows/wsl/about">Windows Subsystem for Linux</a> a go to see how far I could get with it. Trying to get Hadoop to run was as good a test as any. It turns out it wasn’t all that difficult and I’m far from a Linux expert.</p>
<p>If you want to reproduce my results, follow along with the official Single Cluster <a href="https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-common/SingleCluster.html">getting started doc</a> on the <a href="https://hadoop/apache.org">Hadoop site</a>. Make sure you use the one for the version you want to install or else you’ll stumble over things like the port that the namenode admin site runs on (50070 with Hadoop 2 and 9870 for 3). I chose 3.1.0.</p>
<p>Installing WSL is as easy as going through the Windows store and selecting the distribution of your choice. I chose Ubuntu 18.04. Once WSL is installed, you need to make it usable. The color scheme is a monstrosity of illegibility. Here are some good <a href="https://medium.com/@iraklis/fixing-dark-blue-colors-on-windows-10-ubuntu-bash-c6b009f8b97c">instructions</a> on how to get to something that won’t make your eyes bleed.</p>
<p>You’ll be spending some time in Vi, so let’s make that useable to. Add the following line to ~/.vimrc, or else you’ll be squinting at an illegible dark blue on black color scheme:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">set </span><span class="nv">background</span><span class="o">=</span>dark
</code></pre></div></div>
<p>Now, let’s get to what I discovered. Below I’m taking the approach of filling in the gaps of the ‘Getting Started’ doc. Reading this post first should (hopefully) get you over the hurdles you’ll run into while you’re following the doc.</p>
<h2 id="installing-dependencies">Installing dependencies</h2>
<p>Hadoop’s main dependency is Java, which is not installed by default in WSL. <a href="https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04">Here</a> are some straight forward instructions on how to install it. Note that your JAVA_HOME will almost certainly be different than the Getting Started docs. The best way I know how to check is to run:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>update-alternatives <span class="nt">--config</span> java
</code></pre></div></div>
<p>That will show you something like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/usr/lib/jvm/java-8-oracle/jre/bin/java
</code></pre></div></div>
<p>So, for me, JAVA_HOME should be set to <code class="language-plaintext highlighter-rouge">/usr/lib/jvm/java-8-oracle/</code> in <code class="language-plaintext highlighter-rouge">etc/hadoop/hadoop-env.sh</code>.</p>
<p>Although Hadoop is intended to run on Java 8, that version is getting a little long-in-the-tooth. It is possible to run Hadoop with newer versions of Java, but you will likely run into the following error when loading the administrative site.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Failed to retrieve data from /webhdfs/v1/?op<span class="o">=</span>LISTSTATUS: Server Error
</code></pre></div></div>
<p>As described in this <a href="https://stackoverflow.com/questions/48735869/error-failed-to-retrieve-data-from-webhdfs-v1-op-liststatus-server-error-wh">StackOverflow thread</a>, this is due to deprecated dependency and can be solved by adding the following line to <code class="language-plaintext highlighter-rouge">hadoop-env.sh</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">HADOOP_OPTS</span><span class="o">=</span><span class="s2">"--add-modules java.activation"</span>
</code></pre></div></div>
<h2 id="ssh-issues">SSH issues</h2>
<p>Attempting to connect to localhost via ssh was the only problem I ended up running into. When doing so, I got:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Connection closed by 127.0.0.1 port 22
</code></pre></div></div>
<p>It turned out, the necessary keys for sshd to function properly are not automagically generated in WSL (I think). I’m sure you could go through and generate them yourself, but I found it easier just to reinstall openssh-server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt purge openssh-server
<span class="nb">sudo </span>apt <span class="nb">install </span>openssh-server
<span class="nb">sudo </span>service ssh start
</code></pre></div></div>
<p>And that’s it. If you do the above and then follow along with the ‘Getting Started’ doc, you should be good to go.</p>
<h3 id="issues-starting-hadoop">Issues Starting Hadoop</h3>
<p>When running <code class="language-plaintext highlighter-rouge">start-dfs.sh</code> at one point I got the following error:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Starting namenodes on <span class="o">[</span>localhost]
localhost: jon@localhost: Permission denied <span class="o">(</span>publickey,password<span class="o">)</span><span class="nb">.</span>
Starting datanodes
localhost: jon@localhost: Permission denied <span class="o">(</span>publickey,password<span class="o">)</span><span class="nb">.</span>
Starting secondary namenodes <span class="o">[</span>JON-PC]
JON-PC: jon@jon-pc: Permission denied <span class="o">(</span>publickey,password<span class="o">)</span><span class="nb">.</span>
</code></pre></div></div>
<p>I had deviated from the instructions by adding a password to my ssh key (for added securities!). That was not good for Hadoop. There MUST be a way to start Hadoop with a password protected key, but I haven’t researched that yet. It was easy enough to correct by running the following to remove the password.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-keygen <span class="nt">-p</span>
</code></pre></div></div>I was curious if it was possible to get Hadoop running under Windows Subsystem for Linux. It is.Remembering Peter Lawler2018-05-23T00:00:00+00:002018-05-23T00:00:00+00:00https://jboulineau.github.io/blog/personal/remembering_peter_lawler<p> </p>
<p><center><img src="/assets/images/peter_lawler.jpg" alt="Peter Lawler" /></center></p>
<blockquote>
<p>One year ago today a great man died. To mark the day I’m re-posting what I wrote at the time in memoriam Peter Augustine Lawler.</p>
</blockquote>
<p><em>Originally published May 24th, 2017</em></p>
<p>I’ve spent much of the last two days reading (and re-reading) various eulogies of Peter Augustine Lawler from his friends and colleagues. Many have been touching, most have been accurate, one missed the point completely, and none have been sufficient. I was not Dr. Lawler’s friend and certainly not his colleague, though there was a long period in my life that I hoped to be both. It’s only now that I realize that being his student may have been the much greater and more rewarding privilege. For it is true that he possessed a brilliant mind that was influential to colleagues, and it is true that he was a compassionate man who cared for his friends. Yet, he was more.</p>
<p>The first lesson of the first class I took with Dr. Lawler was the meaning of the word ‘philosophy.’ Dutifully I noted: philia = love, sophia = wisdom. And then we moved on. Over the course of my education, I came to realize that philosophy as practiced is often nothing of the sort. In our time, the common experience of the philosophy student is one of tearing down the very concept of wisdom itself. I also learned that there are those who still hold to the ‘classical’ notion of the efficacy of the exercise of reason. These students of philosophy tended to turn to the past to rediscover thinkers who were, at least on the surface, lovers of wisdom. What made Dr. Lawler distinctive is that he began with the premise that to be wise is first to be honest about being human. It is upon this foundation, not the practices of the classical philosophers, that Peter Lawler based his life. And it is this perspective that must be understood in order to understand the man.</p>
<p>For instance, many have observed his love of popular culture. To some, it appears, this was a quirk of his personality, a quaint peccadillo that stood out amongst the crowd of professional serious thinkers. It was a demonstration of his brilliance that he was able to glean perceptive value from such vulgar sources. To others, it seems to have been perceived as a way of relating to his students, thus highlighting his undeniable devotion to us. Neither of these perspectives are sufficient. For Dr. Lawler, popular culture is the way in which we reveal our collective wandering in a particular moment in time. In my time, for instance, we talked of <em>Fight Club</em>, <em>The Matrix</em>, and <em>Lost in Translation</em> because they were temporally unique expressions of grappling with the experience of human beings immersed in their own alienation.</p>
<p>It is evidence of our neurosis that his take on politics was viewed by some as nonchalant. Nothing could be further from the truth. It is only through the lens of our obsession with the ultimately uninteresting triumph of one fatally flawed ideology over another that Peter Lawler could be viewed as nonchalant. He was too fascinated by the way politics revealed deeper truths about being human – far too fascinated to be overly concerned about who happened to win an election. He did not, after all, choose politics as his academic focus by accident. It was the political philosophers who intrigued him the most because it is in politics, the ultimate cauldron of human relationships, that the strangeness of humanity is laid bare.</p>
<p>His common observation that everything is getting both better and worse at the same time was not the indication of a man unconcerned with the trials and travails of human life or a rejection of the importance of political policy and social action. With this pithy observation that history is a process of progressive regression he was calling us out of our utopian reveries to face the reality that we are wanderers, not conquerors. He was, in fact, constantly engaged in the endeavor to make us face up to our lostness because until we do, until we start from the point of our alienation, our willful efforts will only continue to careen us through a path of accidental successes and failures, which only serves to enhance our malaise. He was showing us that politics (or more precisely, the exercise of power) is not our savior.</p>
<p>He feared no line of thought because of a deep conviction that if we think hard enough, break up into small groups and discuss long enough, that we will end up, after a long journey through the wonderful strangeness of the individual/social animal, at the truth of what it means to be human. His embrace of those who disagreed with him was not the ‘tolerance’ of the relativist, the ‘niceness’ of a good man, or even the noble duty of a teacher, but rather an expression of joy in the act of walking together towards wisdom. He needed no ideological shelter to preserve his thoughts from threats, because it was not his thoughts that centered his life. You can threaten premises upon which argumentation (and academic careers) are built and you can threaten the hopes of political triumphs by challenging ideologies. You cannot threaten love. When you love wisdom you also love those who pursue wisdom and Dr. Lawler had the extraordinary ability to see, even through the most obnoxious of challengers, the searching mind in each of us. He believed more deeply in our capability to encounter Truth than anyone I’ve ever met. He was, in fact, a radical.</p>
<p>This leads me to being his student. All of the above was discoverable by colleagues, if they read closely enough, or friends, if they asked the right questions (and, perhaps, became students themselves). But, to be his student was not just to know him, but to experience him in practice. It is the nature of love to spread love and it was with his students that his efforts could bear the most fruit. So I have to disagree with the eulogist who said that the best example of Dr. Lawler’s thought was Walker Percy’s <em>Lost in the Cosmos</em>. In fact, his thought was best expressed in each moment of the classroom, communicated in the very act of teaching itself. To be taught by Peter Lawler was to be pushed, pulled, yanked, cajoled, encouraged, corrected, and all-out willed into loving wisdom. And, it turns out, you cannot become wise unless you first learn, above all, to love humanity in all its strangeness; to see humanity as God sees humanity. What he taught was not simply a body of philosophical knowledge, or even how to think deeply about the most important questions. He taught us how to live well. And if you go home and think about that the rest of the day you’ll realize that’s the best you can say about anyone.</p>
<p>A+/A+</p>One year ago today a great man died. To mark the day I'm re-posting what I wrote at the time in memoriam Peter Augustine Lawler.Lambda Architecture2018-05-21T00:00:00+00:002018-05-21T00:00:00+00:00https://jboulineau.github.io/blog/architecture/lambda-architecture<p>In the last part of the series we (nearly) completed the design of a conceptual real-time data streaming application, but were left with two remaining issues: <strong>shrinking batch windows</strong> and <strong>processing inefficiency</strong>. In large part the IT community has settled upon the solution: fault-tolerant compute clusters and distributed file systems, lead primarily by the Hadoop ecosystem and popular NoSQL data stores like MongoDB. We can mitigate the two primary challenges faced when working with shrinking batch windows, processing failures and data size, by scaling processing across multiple computers and building in (some types of) fault-tolerance to the system. As our batch windows shrink and data size grows the simplest and (often) most cost effective solution is to add commodity hardware to a compute cluster. However, these systems come with significant complexity. It is in this context that the Lambda Architecture was conceived.</p>
<p>The Lambda<sup>1</sup> Architecture was defined in a <a href="http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html">2011 blog post by Nathan Marz</a> and further detailed in his book, <a href="https://www.manning.com/books/big-data">Big Data</a>. The (perhaps regrettable) title of his post, <em>How to Beat the CAP Theorem</em>, shows that he was concerned with the fundamental tradeoffs of distributed systems: consistency, availability, and partition tolerance (which you <a href="https://codahale.com/you-cant-sacrifice-partition-tolerance/">cannot sacrifice</a>). In other words, while this series approaches the subject from the perspective of the batch ETL developer, Marz was attempting to overcome problems with the real-time processing systems circa 2011. A lot has been discussed about the CAP Theorem, much of which exposed the misconceptions and limited usefulness of the theorem as defined (see <a href="http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html">Problems with CAP, and Yahoo’s little known NoSQL system</a> by Daniel Abadi (<a href="https://twitter.com/daniel_abadi">T</a>) and especially <a href="https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html">Please stop calling databases CP or AP</a> by Martin Kleppmann (<a href="https://twitter.com/martinkl">T</a>) ). Nevertheless, it remains a useful, if not sufficient, way to reason about distributed systems. Marz was wrestling with an essential question: given the reality of unreliable technological and human systems, how can we meet the potentially competing requirements of producing consistent and accurate data processing results in real-time?</p>
<p>Marz described an architecture of ‘layers’, the ‘speed layer,’ ‘batch layer,’ and ‘serving layer,’ as shown in the following figure:</p>
<p><center><img src="/assets/images/lambda.png" alt="Lambda Architecture" /></center></p>
<p><a href="https://dzone.com/articles/lambda-architecture-with-apache-spark">Image source</a></p>
<p>The batch layer is composed, as you would expect, of a distributed file system such as HDFS and a computation engine like Spark. In order to achieve real-time capabilities, however, another set of systems must be included. A low-latency distributed data store and a built-for-purpose real-time computation engine such as the Marz-authored <a href="http://storm.apache.org/">Apache Storm</a>, which is a popular option, are also required.</p>
<p>The concept is this: the batch layer provides the foundation of the architecture by producing provably-accurate results at a point-in-time at scale. In the time between execution of batch processing, the ‘speed layer’ is to consume data as it arrives and provide approximate calculations. These calculations may suffer from inaccuracies common to real-time processing, such as those caused by out-of-order messages, mistakes in code, or changing business requirements. But, not to worry, these temporary inaccuracies are corrected by the batch layer the next time processing occurs. Upon completion of batch processing, accumulated data that is not required for stateful real-time computation is purged from the speed layer, which minimizes resource utilization and latency. Thus, the ‘speed layer’ exists simply to bridge the timeliness gap between batch processing windows. Batch and real-time functions produce separate materialized views of the data that must be unioned at the ‘serving layer.’ The ‘serving layer’ is responsible for ensuring that the results produced by the two computation layers are reconciled and presented in a semantically consistent manner.</p>
<p>For Marz, batch processing presents a solution to the problems of real-time distributed data processing. At least in the presence of a system like Hadoop, it is as scalable. Most importantly, it is as simple data processing gets. Apply logic to a set of unchanging data and receive (eventually) accurate results. In his context wherein traditional batch processing problems have been solved through modifications to the overall architecture as described in parts <a href="/blog/architecture/a-simple-real-time-app">4</a> and <a href="/blog/architecture/improving-the-real-time-app">5</a> of this series Marz is correct that the primary problem with batch processing is that it does not produce results in real-time.</p>
<p>The Lambda Architecture has been implemented with success many times. And this is, needless to say, only a very high level overview appropriate for an introduction. I strongly urge you to read the <a href="http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html">blog post</a> at a minimum, or even the book. It is, however, not without flaws. In the next part of the series we will examine these flaws in context of one proposed alternative, the <em>Kappa Architecture</em>.</p>
<hr />
<p> </p>
<ol>
<li>The term <em>Lambda</em>, refers to the computer science concept of <a href="https://en.wikipedia.org/wiki/Anonymous_function">anonymous functions</a> that emerges from <a href="https://en.wikipedia.org/wiki/Lambda_calculus">Lambda calculus</a>. In this context, the fundamental principle of the architecture is to apply arbitrary functions to the data regardless of the layer of processing operations.</li>
</ol>Through the first parts in this series we have covered problems with batch ETL processes and conceptually designed a real-time data processing system. In this post the series shifts to looking at reference architectures that have been successfully used to implement real-time data streaming solutions. The first of these is known as the Lambda Architecture.