Process AWS Kinesis Firehose data with Python

Pipelining streamed data events directly into Amazon S3 via the AWS Kinesis Firehose service is convenient and efficient. And we could use Python with boto3 to consume the data directly from S3. This allows for seamless storage of your data, ensuring its integration and accessibility. Mostly, we are dealing with JSON-formatted event logs. But there is one tiny stone in the shoe for logs feeding from AWS Kinesis Firehose, there is no newline between consecutive log entries. Continue reading

Thoughts on the fading away of the age of IE

Nowadays, as it pleased most frontend engineers, making web pages/applications compatible with IE Browser is not required almost everywhere. Internet Explorer, especially the lower versions, is inconsistent with W3C standards, barely has developer toolchains, and therefore very hard to debug. But the anti-modern frontend legacy browsers didn’t easily die. If you had checked browser distribution in recent years, you would notice that IE always had a reasonable proportion, not much, but also not in a trend towards zero in a short period. Continue reading

The curious cases of json_extract

json_extract is a function for extract data from a JSON document both in MySQL and MariaDB. The function normally works fine, for example set @j = '{"num": 42, "list": [1, 2, 3], "obj": {"name": "Edward Stark"}}'; select json_extract(@j, '$.num') as num, json_extract(@j, '$.list') as list, json_extract(@j, '$.obj') as obj; +------+-----------+--------------------------+ | num | list | obj | +------+-----------+--------------------------+ | 42 | [1, 2, 3] | {"name": "Edward Stark"} | +------+-----------+--------------------------+ But when it comes to a single JSON string, the results of json_extract is not always as expected. Continue reading

Contextvars and Thread local

Here in the post, I will share some examples about the contextvars (new in Python 3.7) and thread local. Default Value In the module level, use ContextVar.set or directly setattr for a thread local variable, won’t successfully set a default value, the value set won’t take effect in another thread. To ensure a default value, for contextvars import contextvars context_var = contextvars.ContextVar("context_var", default=0) for thread local, a sub class of thread. Continue reading

The Tiny Chips from Chinese Hackers: When Falsifiability meets Public Perception

During the 2018 Chinese National Day Holidays, Bloomberg Business Week reported that Chinese hackers planted microchips into motherboards supplying for data center servers of tech giants Apple and Amazon. The original Bloomberg post is available at https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies, and Amazon, Apple, Super Micro and the Chinese Government all have denied the hack. As I followed the related posts, I found despite there may(or may not) be a hack on the motherboard chips, surely the event happening and growing is a big hack on the public perception. Continue reading

The Ponzi Software Development Scheme

The metaphor Ponzi Software Development Scheme, came to me after I have been read about a post from CACM, The Death of Big Software. The traditional big softwares will die away, because they are easily growing to become too hard to maintain, and will be replaced by cloud or microservices architecture based softwares. But will cloud or microservices save big software projects from failure? I rather say, no silver bullets, neither cloud nor microservices. Continue reading