For the sake of honest comparison, ElasticSearch repo has jaw-dropping 1076 PRs merged for the same month, and in terms of features, their pace is very impressive, as well!

We are using Clickhouse for log storage and analytics in ApiRoad.net project (which is an API marketplace where developers sell their APIs, still in active development) and we are happy with the results so far. As an API developer myself, I know how important is the observability and analysis of HTTP request/response cycle to maintain the quality of service and quickly detect bugs, this is especially true for pure API service. (If you are an API author and want to utilize ApiRoad analytics & billing plaftform to sell API subscriptions, drop me a message at contact@apiroad.net with your API description – I will be happy to chat!)

We are also using ELK (ElasticSearch, Logstash, filebeat, Kibana)  stack on other projects, for very similar purposes - getting http and mail logs, for later analysis and search via Kibana.

And, of course, we use MySQL. Everywhere!

This post is about the major reasons why we chose Clickhouse and not ElasticSearch (or MySQL) as a storage solution for ApiRoad.net essential data - request logs (Important note: we still use MySQL there, for OLTP purposes).

1. SQL support, JSON and Arrays as first class citizens.

SQL is a perfect language for analytics. I love SQL query language and SQL schema is a perfect example of boring tech that I recommend to use as a source of truth for all the data in 99% of projects: if the project code is not perfect, you can improve it relatively easily if your database state is strongly structured. If your database state is a huge JSON blob (NoSQL) and no-one can fully grasp the structure of this data, this refactoring usually gets much more problematic.

I saw this happening, especially in older projects with MongoDB, where every new analytics report and every new refactoring involving data migration is a big pain. Starting such projects is fun – as you don't need to spend your time carefully designing the complete project schema, just "see how it goes" – but maintaining them is not fun!

But, it is important to note that this rule of thumb - "use strict schema" - is not that critical for log storage use cases. That's why ElasticSearch is so successful, it has many strong sides, and flexible schema.

Back to JSON: traditional RDBMS are still catching up with NoSQL DBMS in terms of JSON querying and syntax, and we should admit JSON is a very convenient format for dynamic structures (like log storage).

Clickhouse is a modern engine that was designed and built when JSON was already a thing (unlike MySQL and Postgres), and Clickhouse does not have to carry the luggage of backward compatibility and strict SQL standards of these super-popular RDBMS, so Clickhouse team can move fast in terms of features and improvements, and they indeed move fast. Developers of Clickhouse had more opportunities to hit a sweet balance between strict relative schemas and JSON flexibility, and I think they did a good job here. Clickhouse tries to compete with Google Big Query and other big players in the analytics field, so it got many improvements over "standard" SQL, which makes its syntax a killer combo and in a lot of cases much better than you get in traditional RDBMS, for analytics and various calculation purposes.

Some basic examples:

In MySQL, you can extract json fields, but complex JSON processing, like joining relational data on JSON data, became available only recently, from version 8 with JSON_TABLE function. In PosgreSQL, the situation is even worse - no direct JSON_TABLE alternative until PostgreSQL 12!

Compare it to Clickhouse JSON and related arrays feature set - it is just miles ahead. Links:

• arrayJoin
• groupArray
• arrayMap
• arrayFilter

These are useful in a lot of cases where you would use generate_series() in PostgreSQL. A concrete example from ApiRoad: we need to map requests amount on chart.js timeline. If you do regular SELECT .. group by day, you will get gaps if some days did not have any queries. And we don't need gaps, we need zeros there, right? This is exactly where generate_series() function is useful in PostgreSQL. In MySQL, the recommendation is to create stub table with calendar and join on it... not too elegant, huh?

Here is how to do it in ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html#_missing_value_2

Regarding the query language: I am still not comfortable with verbosity and approach of ElasticSearch Lucene syntax, HTTP API, and all these  json structures that you need to write to retrieve some data. SQL is my preferred choice.

Here is the Clickhouse solution for dates gap filling:

SELECT a.timePeriod as t, b.count as c from (
	with (select toUInt32(dateDiff('day', [START_DATE], [END_DATE])) ) 
		as diffInTimeUnits
                
	select arrayJoin(arrayMap(x -> (toDate(addDays([START_DATE], x))), 			range(0, diffInTimeUnits+1))) as timePeriod ) a
            
LEFT JOIN 
            
	(select count(*) as count, toDate(toStartOfDay(started_at)) as timePeriod from logs WHERE 
		[CONDITIONS]
		GROUP BY toStartOfDay(started_at)) b on a.timePeriod=b.timePeriod

Here, we generate virtual table via lambda function and loop, and then left join it on  results from logs table grouped by day.

I think arrayJoin + arrayMap + range functions allow more flexibility than generate_series() from Postgres or ElasticSearch approach. There is also WITH FILL keyword available for a more concise syntax.

2. Flexible schema - but strict when you need it

For log storage tasks, the exact data schema often evolves during project lifetime, and ElasticSearch allows you to put huge JSON blob into index and later figure out the field types and indexing part. Clickhouse allows to use the same approach. You can put data to JSON field and filter it relatively quickly, though it won't be quick on terabyte scale. Then, when you see you often need fast query execution on specific data field, you add materialized columns to your logs table, and these columns extract values from existing JSON on-the-fly. This allows much faster queries on terabytes of data.

I recommend this video from Altinity on the topic of JSON vs Tabular schema for log data storage:

3. Storage and Query Efficiency

Clickhouse is very fast in SELECTs, this was discussed in the previous article.

What is interesting, there is a piece of evidence that Clickhouse can be 5-6 times more efficient in storage, comparing to ElasticSearch, while also being literally an order of magnitude faster in terms of queries. Another one (in Russian)

There are no direct benchmarks, at least I could not find any, I believe because Clickhouse and ElasticSearch are very different in terms of query syntax, cache implementations, and their overall nature.

If we talk about MySQL, any imperfect query, missing index, on a table with mere 100 million rows of log data can make your server crawl and swap, MySQL is not really suited for large-scale log queries. But, in terms of storage, compressed InnoDB tables are surprisingly not that bad. Of course, it's much worse in terms of compression comparing to Clickhouse (sorry, no URLs to benchmarks to support the claim this time), due to its row-based nature, but it still often manages to reduce cost significantly without a big performance hit. We use compressed InnoDB tables for some cases for small-scale log purposes.

4. Statistics functions

Getting median and .99 percentile latency of 404 queries is easy in Clickhouse:

SELECT count(*) as cnt, 
  quantileTiming(0.5)(duration) as duration_median, 
  quantileTiming(0.9)(duration) as duration_90th, 
  quantileTiming(0.99)(duration) as duration_99th
  FROM logs WHERE status=404

Notice usage of quantileTiming function and how currying is elegantly used here. Clickhouse has generic quantile function! But quantileTiming is optimized for working with sequences which describe distributions like loading web pages times or backend response times.

There are more than that. Want weighted arithmetic mean? Want to calculate linear regression? this is easy, just use specialized function.

Here is a full list of statistics functions of Clickhouse:

https://clickhouse.tech/docs/en/sql-reference/aggregate-functions/reference/

Most of these are problematic to get in MySQL.

ElasticSearch is much better in this than MySQL, it has both quantiles and weighted medians, but it still does not have linear regression.

5. MySQL and Clickhouse tight integration

MySQL and Clickhouse has integrations on multiple levels, which make it easy to use them together with minimum of data duplication:

• MySQL external dicts
• MySQL database replica inside Clickhouse (via binlog)
• MySQL database engine - similar as previous one but dynamic, without binlog
• MySQL table function to connect to MySQL table in specific SELECT query
• MySQL table engine to describe specific table statically in CREATE TABLE statement
• Clickhouse can speak MySQL protocol

I can't say for sure how fast and stable dynamic database engines and table engines work on JOINs, this definitely requires benchmarks, but the concept is very appealing - you have full up-to-date clone of your MySQL tables on your Clickhouse database, and you don't have to deal with cache invalidation and reindexing.

Regarding using MySQL with Elasticsearch, my limited experience says that these two techonologies are just too different and my impression is that they are speaking foreign languages, and do no play "together", so what I usually did is just JSONify all my data that I needed to index in ElasticSearch, and send it to ElasticSearch.  Then, after some migration or any other UPDATE/REPLACE happen on MySQL data, I try to figure out the re-indexing part on Elasticseach side. Here is an article of the Logstash powered approach to sync MySQL and ElasticSearch. I should say I don't really enjoy Logstash for it's mediocre performance, and RAM requirements, and since it is another moving part which can break. This syncing and re-indexing task is often a significant stop factor for us to use Elasticsearch in simple projects with MySQL.

6. New Features

Want to attach S3 stored CSV and treat it as table in Clickhouse? Easy.

Want to update or delete log rows to be compilant with GDPR? Now, this is easy!

There was no clean way to delete or update data in Clickhouse in 2018 when my first article was written, and it was a real downside. Now, it's not an issue anymore. Clickhouse leverages custom SQL syntax to delete rows:

ALTER TABLE [db.]table [ON CLUSTER cluster] DELETE WHERE filter_expr

This is implemented like this to be explicit that deleting is still a pretty expensive operation for Clickhouse (and other columnar databases) and you should not do it every second on production.

7. Cons

There are cons for Clickhouse, comparing to ElasticSearch. First of all, if you build internal analytics for log storage, you do want to get the best GUI tool out there. And Kibana is good nowadays for this purpose when you compare it to Grafana (at least, this point of view is very popular on the Internet, Grafana UI is not that slick sometimes). And you have to stick to Grafana or Redash if you use Clickhouse. (Metabase, which we adore, also got Clickhouse support!)

But, in our case, in ApiRoad.net project, we are building customer-facing analytics, so we have to build analytics GUI from scratch, anyways (we are using a wonderful stack of Laravel, Inertia.js, Vue.js, and Charts.js to implement the customer portal, by the way).

Another issue, related to the ecosystem: the selection of tools to consume, process data and send them to Clickhouse is somewhat limited. For Elasticsearch, there are Logstash and filebeat, tools native to Elastic ecosystem, and designed to work fine together. Luckily, Logstash can also be used to put data to Clickhouse, this mitigates the issue. In ApiRoad, we are using our own custom-built Node.js log shipper which aggregates logs and then sends them to Clickhouse in a batch (because Clickhouse likes big batches and does not like small INSERTs).

What I don't like in Clickhouse is also weird naming of some functions, which are there because Clickhouse was created for Yandex.Metrika (Google Analytics competitor), e.g. visitParamHas() is a function to check if a key exists in JSON. Generic purpose, bad non-generic name. I should mention that there is a bunch of fresh JSON functions with good names: e.g. JSONHas(), with one interesting detail: they are using different JSON parsing engine, more standards-compliant but a bit slower, as far as I understand.

Conclusion

ElasticSearch is a very powerful solution, but I think its strongest side is still huge setups with 10+ nodes, used for large-scale full-text search and facets, complex indexing, and score calculation – this is where ElasticSearch shines. When we talk about time-series and log storage, my feeling is there are better solutions, and Clickhouse is one of them. ElasticSearch API is enormous, and in a lot of cases it's hard to remember how to do one exact thing without copypasting the exact HTTP request from the documentation, it just feels "enterprisy" and "Java-flavored". Both Clickhouse and ElasticSearch are memory hungry apps, but RAM requirements for minimal Clickhouse production installation is 4GB, and for ElasticSearch it is around 16GB. I also think Elastic team focus is getting pretty wide and blurred with all the new amazing machine-learning features they deploy, my humble opinion is that, while these features sound very modern and trendy, this enormous feature set is just impossible to support and improve, no matter how many devs and money you have, so ElasticSearch more and more gets into "Jack of all trades, master of none" category for me. Maybe I am wrong.

Clickhouse just feels different. Setup is easy. SQL is easy. Console client is wonderful. Everything just feels so light and makes sense, even for smaller setups, but rich features, replicas, and shards for terabytes of data are there when you need it.

Good external links with further info on Clickhouse:

Altinity Blog

https://blog.luisico.net/2019/03/17/testing_clickhouse_as_logs_analysis_storage/

UPD: this post hit top#1 on HackerNews, useful comments there, as usual!

Best comments:

ClickHouse is incredible. It has also replaced a large, expensive and slow Elasticsearch cluster at Contentsquare. We are actually starting an internal team to improve it and upstream patches, email me if interested!

I'm happy that more people are "discovering" ClickHouse. ClickHouse is an outstanding product, with great capabilities that serve a wide array of big data use cases. It's simple to deploy, simple to operate, simple to ingest large amounts of data, simple to scale, and simple to query. We've been using ClickHouse to handle 100's of TB of data for workloads that require ranking on multi-dimensional timeseries aggregations, and we can resolve most complex queries in less than 500ms under load.

Also from HN:

It turns out Uber just switched from ELK to Clickhouse for log analytics, read more on their writeup.

Another way would be to go for paid solutions, like elfsight widget ( https://elfsight.com/instagram-feed-instashow/ ) but their artificial restriction on amount of views on the free and $5 plan is just crazy.

So, I've found a nice alternative, which is much cheaper (literally, free for fair use), does not require Instagram developer account, has open source code, and is very customizable: https://github.com/restyler/inwidget

The demo website with sample widgets: https://inwidget.apiroad.net/

This library still requires subscription for a cloud proxy (the same RapidAPI solution I told you about on previous post),  but for the end user, this is not an issue - just cache the result on the PHP side, refresh the cache every 24 hours, and  easily stay under the free plan of the API!

The best part of the inWidget proxified library, besides its stable work, is that the template of a widget is just plain html and php, so it is easy to customize the output, even if for a newbie in programming.

1. Forever

URL: https://github.com/foreversd/forever

Github stars: 13.2k

Pros: super simple and minimal.  

Cons: not active in development. Heavily tied to node.js eco system. README of the project recommends to use pm2 or nodemon.

Conclusion:

This is probably not a good solution for new projects, though it's pure JS and is pretty minimal, which I like.

2. PM2

URL: https://github.com/Unitech/pm2

Github Stars: 34k

Pros:

• A lot of features.
• Specializes in Node.js.
• Zero downtime reload feature is a big one!  

Cons:

• Big amount if features sometimes mean more cognitive overhead to run when you need super-basic setup, also more moving parts and higher chance that something may break.
• Is actively monetized so the solution includes paid features.  For example, metrics is a paid feature.
• Tied to node.js eco system.

Conclusion:  

Jack-of-all-trades. This is a very good solution if you develop only node.js apps, like a lot of features out of the box, and are okay that it way try to monetize on you.

3. Nodemon

URL: https://github.com/remy/nodemon/

Github Stars: 21.8k

Pros:

• Boring tech: old & proven
• Unix-like simplicity: does one thing and does it right

Cons:

• This is not exactly  intended to be used in production.
• Tied to node.js eco system

Conclusion:

I use nodemon every day, but not for production purposes, since its main purpose is to do reload when you save a file, so your code changes are applied to living process. But I know some developers use if for production as well. I would say you should use it if you develop only node apps and you liked forever.js brutal simplicity.

4. Supervisor

URL: https://github.com/Supervisor/supervisor

Github Stars: 6.6k

Pros:

• generic: useful not just for node.js - official way to daemonize Laravel queues
• feature-rich but in a good way - no excess or paid features
• convenient to use
• boring tech

Cons:

• Python (great language but not exactly performant)
• Not very trendy on Github (because it is boring, right?)

Conclusion:

Supervisor is my current favorite. Mostly because it's generic enough to work with Laravel and node.js processes at the same time, has basic logging out of the box, and has multiple processes setup, too. You don't need big performance for this kind of software, at the same time developer ergonomics of Supervisor is just great.

5. Docker

URL: https://github.com/docker

Docker is definitely not a process manager, and is a too big topic for this post, but it solves a lot of things that we used to solve using process managers, in a beautiful way, so take a look at it when you have time (a spare week.. or two).

Pros:

• Is very powerful and has separate ecosystem
• Huge momentum: everyone is dockerizing everything now
• Generic: can daemonize your node.js process or your database

Cons:

• It is a separate world of complexity and overhead. If you want to quickly prototype and daemonize small JS script this is an overkill

Conclusion:

A lot of people say that PM2 is not needed anymore because of Docker.

I use Docker for a lot of things, but its cognitive overhead is significant. Definitely take a look at it, anyways.

I've spent tens of hours on Docker recently, and I really like the concept, I use it in production pretty successfully, too, but it definitely requires a lot of time and context switching to write Dockerfiles,  build images and test everything - this is getting especially tough when you switch back and forth between 3-4 small projects which quickly evolve. I also don't like the fact that Docker eats a lot of CPU on my fresh Macbook 16" even when it does nothing. VS code remote mitigates this, but not completely, it  is still not perfect for active development.

The main selling point of Docker is that it allows you to build 100% reproducible environment. This is a precious gift when you work in a team, but for sole developer in a side project, this is not always needed, and remember: you have to pay for Docker with your time and context switching, and I felt that it was regularly breaking my flow in smaller projects.

So, I definitely use Docker, but I prefer to use it for other guys projects, to run it on my server, or for my projects which are pretty stable and has multiple developers working on it. In this case, you get all the pros, and almost no cons :)

6. systemd

SystemD is a process manager that you don't need to install (okay, this is true only if you use Ubuntu or Debian - but it is true for me), and it's the best thing about it.

Pros:

• Is already installed on your droplet
• Fast

Cons:

• Developer ergonomics is not perfect: I need to google every time to find out where should I put my systemd init file and how to apply it so it works fine.

I've been using systemd for a few years, but no I am gradually switching back to supervisor, because it is very similar, but is more feature rich, and it provides better developer ergonomics in some parts.  The config file for systemd can be generated via online services like https://mysystemd.talos.sh/ and https://techoverflow.net/2019/03/11/simple-online-systemd-service-generator/

Systemd is a good, generic and minimalistic approach if you work on Linux-based systems without Docker setup, but it was not working perfectly with Laravel multiple workers ( https://github.com/laravel/ideas/issues/1570#issuecomment-475499303 ) so I decided to replace it with Supervisor as my go-to approach.

My approach to daemonizing node.js processes

I try to choose and stick to one process manager on all my new projects, if it is possible, to lessen the cognitive overhead when you log in into some production server and try to quickly debug & restart some process.

Here is what I use currently for all new projects (in 2021):

• supervisor for production
• nodemon for development

This was a natural choice because I develop Laravel and Node.js apps.

Clickhouse is an open source column-oriented database management system built by Yandex. Clickhouse is used by Yandex, CloudFlare, VK.com, Badoo and other teams across the world, for really big amounts of data (thousands of row inserts per second, petabytes of data stored on disk).

Qwintry started using Clickhouse in 2018 for reporting needs, and it deeply impressed us by its simplicity, scalability, SQL support, and speed. It is so fast that it looks like magic.

Simplicity

Clickhouse is installed by 1 command in Ubuntu.
If you know SQL - you can start using Clickhouse in no time. It does not mean you can do "show create table" in MySQL and copy-paste SQL to Clickhouse, though.
There are substantial data type differences on table schema definitions, comparing to MySQL, so it will require some time to alter table definitions and explore table engines to feel comfortable.

Clickhouse works great without any additional software, but ZooKeeper needs to be installed if you want to use replication.

Analyzing the performance of queries feels good - system tables contain all the information and all the data can be retrieved via old and boring SQL.

Performance

Benchmark against Vertica and MySQL

Cloudflare post about Clickhouse

Benchmark against Amazon RedShift [2]

Maturity

Clickhouse development happens on Github repo, at an impressive pace.

Popularity

Clickhouse popularity seems to grow exponentially, especially in the Russian-speaking community. Recent Highload 2018 conference (Moscow, 8-9 nov 2018) showed that such monsters as vk.com and Badoo are using Clickhouse in production and inserting data (e.g. logs) from tens of thousands of servers simultaneously ( https://www.youtube.com/watch?v=pbbcMcrQoXw - sorry, Russian language only).

Use cases

After I've spent some time on research I think there is a number of niches where Clickhouse can be useful and may even replace other, more traditional and popular solutions:

Augment MySQL and PostgreSQL

We've just replaced (partially) MySQL with ClickHouse for Mautic newsletter platform ( https://www.mautic.org/ ) which due to (questionable) design choices logs each email it sends, and every link in this email with big base64 hash to the huge MySQL table (email_stats). After sending mere 10 million emails to our subscribers this table easily takes 150GB of file space and MySQL starts to feel bad on simple queries. To fix the file space issue we successfully used InnoDB table compression which made the table 4 times smaller, but it still does not make a lot of sense to store more than 20-30 millions of emails in MySQL just for read-only historical information, since any simple query that for some reason has to do full scan leads to swapping and big I/O load - so we were getting Zabbix alerts all the time.

Clickhouse uses two compression algorithms, and typically the compression is closer to 3-4 times but is this specific case the data was very compressible.

Replace ELK

From my experience, ELK stack (Elasticsearch in particular) takes far more resources to run than it should when we talk about log storage purposes. Elasticsearch is a great engine if you need good full-text search (do you need full text search on your logs? I don't), but I wonder why it became de-facto standard for logging purposes - its ingestion performance, combined with Logstash, gave us troubles even on pretty small loads, and required adding more and more RAM and disk space. Clickhouse as a DB layer is better than ElasticSearch:

• SQL dialect support
• Much better compression of stored data
• Regex search support instead of full text
• Better query plans and overall performance

The biggest issue that I see now with Clickhouse (vs ELK) is the lack of log shipping solutions and documentation/tutorials on this topic (everyone can setup ELK by Digital Ocean manual, which is a huge thing for rapid technology adoption). So, the DB engine is here, but there is no Filebeat for Clickhouse yet. Yes, there are fluentd and loghouse, and there is https://github.com/Altinity/clicktail, but more time is required so the simple and best way takes the strong lead so newcomers just install and use it in 10 minutes.
Since I like minimalistic solutions I tried to use FluentBit (which is a log shipper with very small memory footprint) with Clickhouse (and I tried to avoid using Kafka in between) but small incompatibilities like date format issues still need to be figured out before this can be done without proxy layer which converts data from Fluentbit to Clickhouse.

Talking about Kibana alternative - there is Grafana which can be used with Clickhouse as backend. As far as I understand there may be issues with performance when rendering huge amount of data points, especially with older Grafana versions - at Qwintry we haven't tried it yet, but complaints about this appear from time to time in Clickhouse telegram support channel.

Replace Google Big Query and Amazon RedShift (for bigger companies)

A perfect use case for Big Query is: upload 1TB of JSON data and run analytical queries on it. Big Query is a great product and it's hard to overestimate its scalability. It is a lot more sophisticated piece of software then Clickhouse running on an in-house cluster, but from the customer point of view, it has a lot of similarities with Clickhouse. Big Query can quickly get pricey since you pay for each SELECT, and it is a SaaS solution with all its pros and cons.
Clickhouse is a great fit if you run a lot of computationally expensive queries. The more SELECT queries you run every day - the more sense it has to replace Big Query with Clickhouse - it may literally save you thousands of dollars if we talk about many terabytes of processed data (not stored data, which is pretty cheap in Big Query).

Altinity summed it up well in "Cost of Ownership" section in their article.

Replace TimescaleDB

TimescaleDB is a PostgreSQL extension, specializing in timeseries data.
https://docs.timescale.com/v1.0/introduction

Clickhouse did not really start to seriously compete in the time-series niche, but due to its columnar nature and vectorized query execution it is faster than TimescaleDB in most of the analytical queries, batch data ingestion performance is ~3x better  and Clickhouse uses 20 times less disk space which is really important for big amounts of historical data:
https://www.altinity.com/blog/clickhouse-for-time-series

The only way to save some disk space when using TimescaleDB is to use ZFS or similar file system.

Upcoming updates to Clickhouse will most likely introduce delta compression which will make it even better fit for time-series data.

TimescaleDB may be a better choice than (bare) Clickhouse for:

• small installations with very low amount of RAM (<3 GB),
• big amount of frequent small INSERTs which you don't want to buffer into bigger chunks.
• better consistency and ACID
• PostGIS support
• tight mix and easy joins with existing PostgreSQL tables (since essentially TimescaleDB is PostgreSQL)

Compete with Hadoop and MapReduce systems

Hadoop and other MapReduce products can do a lot of complex calculations, but these tend to have huge latencies, and Clickhouse fixes it - it processes terabytes of data and gives results almost instantly. So for rapid, interactive analytical research Clickhouse can be a lot more interesting for data engineers.

Compete with Pinot and Druid

Closest contenders (column-oriented, linearly scalable, open source) are Pinot and Druid, there is a wonderful write-up with comparison here:
https://medium.com/@leventov/comparison-of-the-open-source-olap-systems-for-big-data-clickhouse-druid-and-pinot-8e042a5ed1c7
The article is a bit updated: it states that Clickhouse does not support UPDATE and DELETE operations, which is not exactly the case for latest versions.

We do not have any experience with these, but
I don't like the complexity of infrastructure that is required to run Druid and Pinot  - big amount of moving parts, and Java everywhere.

Druid and Pinot are Apache incubator projects and their GitHub pulse pages show good development pace. BTW, Pinot went for Apache Incubator just a month ago - in October of 2018. Druid went for Apache Incubator just 8 months earlier on 2018-02-28.

This is really interesting and raises some (stupid?) questions because I have very little information on how ASF works.
Did Pinot authors notice that Apache Foundation is giving a lot of acceleration to Druid and felt a bit envious? :)
Will it somewhat slow down Druid and accelerate Pinot development (of course if there is a phenomenon of volunteer contributors which were committing to Druid but suddenly got interested in Pinot)?

Clickhouse CONS

Immaturity

This is obviously not a boring technology yet (but there are no such thing in columnar DBMS, anyways)

Small inserts at high rate perform poorly

Inserts need to be batched into bigger chunks, the performance of small inserts degrades proportionally to a number of columns in each row. This is just how data is stored on disk in Clickhouse - each column means 1 file or more, so to do 1-row insert containing 100 columns at least 100 files needs to be opened and written to. That's why some mediator is required to buffer inserts (unless the client can't buffer it) - usually, Kafka or some queue system. Or, Buffer table engine can be used and data can be copied to MergeTree tables later in bigger chunks.

Joins are restricted by server RAM

Well, at least there are joins! E.g. Druid and Pinot do not have joins at all - since they are hard to implement right in distributed systems, which do not support moving big chunks of data between nodes.

Conclusion

At Qwintry, we plan to use Clickhouse extensively in the upcoming years, because it hits a great balance of performance, low overhead, scalability, and simplicity. I am pretty sure its adoption will increase rapidly as soon as Clickhouse community generates more recipes on how it can be used at small and medium-sized installations.

It is designed to die after execution.

You should not fear code duplication, or bad formatting of code, of bad variables naming - mentioned artefacts of code smell are bad, but are nowhere close to statefulness in terms of contaminating your code.

One of the biggest restrictions of PHP, hated by a lot of people, and at the same time the single and the only huge factor which make poor PHP code written by junior developers manageable is this. State in most cases is completely separated from PHP process and is stored in (server-side) database and even in shittiest code samples the database is something you can rely on as on the source of truth, so reproducing the bugs is (often) easy, and gradual code rewriting (old code and new code using the same database - state) is possible. And the deployment is so easy - you just upload your script! And the development, oh, you just edit this line, save the file... and it works! Think about it.

• Stateless (this is trendy now, right?)
• Easy deployment
• Easy development
• Easy debugging

All of this happiness is just there out of the box, because it dies every time.

Talking about this language design choice - I am even not sure this was intentional (UPD: okay, Rasmus Lerdorf confirmed it was intentional on HN discussion of this article), this might be a side effect. But this restriction, I think, is one of the reasons why PHP, historically full of bad code and strange architectural decisions, gained its popularity, and is still alive as a technology.

This simple concept of strict state separation was alien in JS world because it was not there by design, this is why most frontend frameworks of the first wave (like Angular 1) (and complex jQuery code, of course) were such a pain in the ass before Redux and other solutions were invented, which separated state from interface and finally made it manageable. In JS, you can separate state. In PHP, you have to.

Every time I see Node.js code of sample websocket chat which is a long-living process I can't help myself and shudder because of this global "subscribers" variable which holds all the open websocket connections - I just see how this async long-living complexity explodes in hands of young and fast-typing developer.

This is why Ajax, when it was introduced, while giving so much good stuff to end users, made the life of PHP developers so complex - state suddenly appeared on the frontend and we could not even imagine where this will lead us - jQuery, then Backbone, then Angular, then React, and here we go - now we have the frontend developer profession - separate people dealing with state and stateful interfaces there on the front.

This is why you should be careful when working with queues and daemons (and you will still need them in any mid-sized PHP project). They do not forgive, they are hard to debug and the memory leaks and OS kills them once in a while when your bad code is launched in production. Regular PHP script forgives you, please remember this and appreciate it the next time when you wish your old PHP framework to be better in async code and long-living threads support, or when you praise Docker for being stateless - PHP is stateless, too :)

UPD:
HackerNews discussion: https://news.ycombinator.com/item?id=17853755

I expect Vue to become a primary JS framework in 16-24     months if Evan You makes right steps, at least around     backenders and smaller teams of frontenders. I still consider React stack to be the primary JS framework of 2017, especially if React Native manages to mature and improve itself with the same pace it used to.

me, 10 dec 2016

Since guys from stateofjs.com recently published its 2017 results, there is some material for analysis and thoughts here.

Stateofjs 2017 Frontend Framework results

https://stateofjs.com/2017/front-end/results

http://2016.stateofjs.com/2016/frontend/

So, 1 year passed, and Vue.js is clearly the leader in "would like to learn" by a huge margin which makes me think the next year would be the year of Vue.js success, with React being stable in its growth, while Angular won't be able to keep up with these two rivals. Compare it with 2016 when Vue.js was clearly a dark horse and "another JS framework", Angular was a second choice for "serious guys" and React was a leader.

But.. Vue.js will be dominating only in the web, definitely not in overall frontend world.
React is becoming the technology that will rule the frontend world.

Why?

Synergy and satellite products

https://stateofjs.com/2017/mobile/results/

Synergy, my friends, is the key to React upcoming monopoly.

React Native

Vue.js failed to provide viable alternative to React Native (Weex and Quasar are too young, fragmented and weak), and React Native + React.js is exploding because if you (as an average 2018 developer) master React and Redux for the web, you will get a huge profit right away: if you want, you will be productive in mobile world in a matter of weeks with React Native which clearly demonstrated its advantages over Cordova and other hybrid approaches – just look at Adidas Glitch App, and Skype for Android and iOS, to feel the abyss between React Native and competing hybrid technologies:

https://medium.com/possible-cee/how-we-have-been-breaking-patterns-with-the-adidas-glitch-d734340fd40e

https://mspoweruser.com/skype-is-testing-a-new-android-app-with-a-new-design-reaction-feature-and-bing-integration/

Modern development is about mobile, not just about the web.

React Native is a success, and it will drag React.js to the sky.

We (as a Qwintry team) are preparing big releases of our apps for iOS and Android, scheduled for the first quarter of 2018, for our new website rewritten from scratch (codename Q3, powered by Vue.js and Yii2), and our new apps are powered by React Native.
When we were considering React Native 1.5-2 years ago for our previous versions of apps, our Swift developer conclusion was "definitely no", that was "boring" but logical decision – it was a great idea to wait for technology to mature. We ended up having Swift app for iOS and Java app for Android back then, and I am pretty sure we avoided a lot of pain.

Link to our iOS app which is now written in Swift but will get React Native release in a matter of weeks

Now our Swift developer is writing JS code in React Native and admits it's pretty good and it is the right moment to jump to new stack because advantages of React Native in a lot of cases are now bigger than disadvantages.

Our frontend guys who are writing Vue.js code for web became productive in React Native in a matter of weeks, but I think this process would be even less painful and our stack would be simpler if we chose React.js for the web. We definitely do not regret choosing Vue.js for web, read more in my previous post why we did that, my expectations on Vue.js web domination are becoming the reality, but I still expect a lot of other small and mid-sized teams in 2018 to choose React stack both for web and mobile because of the synergy – it is just an obvious choice now when React Native is so good. Managing separate native mobile development process with Java and Swift/Objective C is still great and safe choice, but it might get expensive from the business point of view, and going through app store approvals still sucks compared to the magic of React Native Codepush.

GraphQL

GraphQL is another product of Facebook which adds value to React ecosystem.
It's not mature yet, it is complex, it has its disadvantages - but it looks like GraphQL is the future replacement for REST.

GraphQL is another sign that innovation in frontend world mostly happens in React universe, and then these innovations are adopted&improved in other frameworks.

More info

Angular is definitely looking "better" here but I don't think this is an indicator that Angular as a framework will be successful in a long-term period – to my mind it is just an indicator that migration from Angular 1 to Angular2+ is an obvious choice for a regular developer who maintains legacy Angular 1 code, but when he tries to migrate and sees the number of changes between old and new versions of Angular – it leads to frustration and this fact explains big number of questions on SO.

https://insights.stackoverflow.com/trends?tags=angularjs%2Cangular%2Creactjs%2Cvue.js%2Cember.js

Could it be different?

There was a moment in 2017 when Vue.js could win the "war" with React in terms of gaining developer traction, even with the absence of proper mobile solution in its stack.
I am talking about the situation with React licensing:
https://ma.tt/2017/09/on-react-and-wordpress/ (HackerNews discussion)

Wordpress was considering to ditch React as the frontend solution for its layout builder, and it could result in Vue.js to be chosen for Wordpress ecosystem pretty much like it was chosen for Laravel.

The intrigue was killed with Facebook just-in-time wise decision to fix the React license:
https://ma.tt/2017/09/facebook-dropping-patent-clause/

Congratulations, Facebook and React team, you did a good job this year, and we will be glad to use React products in our stack, and recommend both Vue.js and React for other teams depending on their situation.

New Qwintry warehouse in New Castle, Delaware, right after Black Friday 2017 customer purchase volumes hit us hard. Every package here is managed, received, and picked via our Swift and React Native powered operator apps, and customers create their orders via Swift/Java (soon React Native) customer apps.

UPD: This post made its way to Hacker News frontpage and there is a useful discussion with 100+ comments there.

• in legacy Drupal system (qwintry.com)
• in our new, completely rewritten qwintry.com branch
• in Yii2-powered b2b system (logistics.qwintry.com)
• in all our smaller internal and external projects (mostly with PHP and Node.js backends)

Our package at customer door - from our happy customer reviews

We have pretty big codebase, mostly PHP&JS.

We decided to use Vue.js after we've completed evaluation of modern frameworks: we've built our customer calculator on React, Vue.js and Angular2.

My thoughts on React.js

React skyrocketed the JS world and it is now probably the default choice for JS devs, when we talk about choosing frontend view framework.
I've built some SPAs and dynamic widgets on React, I've played around React Native (under iOS) and Redux as well. I think that React was a great step forward for JS world in terms of state-awareness, and it showed lots of people the real functional programming in a good, practical way. I think React Native is huge - it just changes the landscape of native development.

Cons of React for me are:

Purity, immutability and ideology over getting things done

Don't get me wrong. I appreciate pure functions and simplistic render() approach - no doubt, that's a great idea which is working great in real life. I am talking about other things.
I guess this level of strictness and purity is something that may be useful when you have 1000 devs in your company - just about the time when you decide to develop your own syntax to go for static types in all the PHP code you write. Or when you are a Haskell developer coming to JS world. But most of companies have far smaller dev teams and other goals than Facebook. I will elaborate more on this below.

JSX sucks

I know, I know! It is "just a plain javascript with special syntax". Our design&html guys who need to focus on making this specific form beautiful by wrapping its elements in various quantities of divs - right now - they don't give a sh*t about purity and plain ES6. Applying designs to React components still sucks big time because JSX lacks readability. Not being able to put plain old IF condition to some block of HTML code sucks, please don't believe React fans that keep telling you that you don't need it when you have ternary operators.  Let me assure you - this is still a mess of HTML and JS when you edit it and read it, even though it gets compiled to pure JS.

<ul>  
       {items.map(item =>
         <li key={item.id}>{item.name}</li>
       )}
</ul>  

Lots of developers (including me - but I am not there anymore) are thinking that this specific restriction on syntax will make you stronger will help you write more modular code because you have to put your chunks of code to smaller helper functions and use them inside your render() function like this guy suggested:
http://stackoverflow.com/a/38231866/1132016

JSX is also the reason when you have to keep splitting your 15-lines-of-html-code component to 3 components, 5-lines-of-code-in-each.

Don't think that this is a great workaround which makes you a better developer because you now have to structure your code like this.

Here is the thing:
When you write a relatively complex component - which you are probably not going to put to public github repo tomorrow to showcase it on hackernews - this approach of splitting components into super-dumb components because of JSX restrictions will always put you out of flow when you are solving real business task. No, I am not saying that the idea of smaller components is bad or not working.
You should clearly realize that you need to split your code into components to keep you codebase manageable and reusable. But you should do it only when you think that this specific logical entity in your code should be a separate component with own props, -  and not on every two-three IFs that you write via ternary operator! Every time you create a new component here and there it costs you and your flow a penny (probably more) because you need to switch from business-task thinking (when you already remember current component state model, and you just need to add some html here and there to make it running) to "manager thinking" - you go create separate file for your component, start thinking about props of this new component, and how they map to state, and how you are going to pass callbacks inside, etc, etc.
As a result you get reduced speed of writing code by being forced to leverage excessive and premature (potential) modularity of components in places where you don't really need it. In my opinion, premature modularity is very similar to premature optimization.

For me and my team the readability of code is important, but it is still very important that writing code is fun. It is not funny to create 6 components when you are implementing really simple calculator widget. In a lot of cases, it is also bad in terms of maintenance, modifications, or applying visual overhaul to some widget, because you need to jump around multiple files/functions and check each small chunk of HTML separately. Again, I am not suggesting to write monoliths - I suggest to use components instead of microcomponents for day-to-day development. It is just about the common sense.

Working with forms and Redux in React will make you type all day long

React is about pureness and clean one way flow, remember? That's why LinkedStateMixin became a persona non grata and now you have to create 10 functions to get input from 10 inputs. 80% of these functions will contain single line with this.setState() call, or redux action call (then you will probably have to create another 10 constants - one per each input). I guess that would be acceptable if you could generate all this code by thinking about it.. but I am not aware of any IDE that could significantly improve this.
Why do you have to type so much? Because two-way binding is considered dangerous by big guys in big-enterprise apps. I can confirm that two-way data flow code sometimes is not as clean to read, but most of these fears are mixed with overall pain about Angular 1 where two-way binding was bad, and still.. probably it was not the biggest fail even there.

I can't share the code for obvious reasons but writing it in Vue was real fun, and the code is very readable.

And I know for sure that creating a separate function for each input to handle a widget like this in React would certainly not make me happy.

Redux sounds like a synonym of verbosity, as well. And it's easy to find developers that blame that Mobx is turning React into Angular just because because it leverages two-way binding - see my point #1 about purity. It looks like a lot of smart people value purity of their codebase more than getting job done (which is fine if you don't have deadlines, I guess).

Excessive tooling

React was created with Babel in mind. You can't do a step in real-world React app without a bunch of npm packages here, compiler to ES5 going first. Simple app based on official react starting package code has around 75MB of JS code in node_modules.
It is not a critical thing, it's more related to JS world in overall than to React, but it adds up to overall frustration when using React as well.

Angular 1: too much freedom is sometimes bad

Angular 1 was a great frontend framework that is located in the opposite corner (from React) of the imaginary JS map of purity and readability of codebases - it allows you to start quickly, it gives you some real fun for your first 1k lines of code, and then it  practically forces you to write shitty code. You will probably get lost in directives, scopes, and two way data flows across all the layers of your app will just be a cherry on the pie of the code that your freshly hired devs won't even want to touch because it won't be manageable.

Why so?
Angular.js was created in 2009 when frontend world looked pretty simple and nobody was even thinking about state hygiene. You can't blame these guys - they were just creating competitor of Backbone with some new concepts and they wanted to do less typing.

Angular2

Just build hello world app and look at the amount of files you got in your repo. You will have to use Typescript (and I am not 100% sure it is something I am going to enjoy doing every day - great writeup from Eric Eliott on this topic) and compilers to start working. It was enough for me.. For me is still too much typing before I start real work. To my mind, Angular 2 guys are trying to build perfect framework which will beat React, instead of trying to build a framework which solves business tasks for average user. May be I am wrong and my mind might change - I don't have a lot of experience building Angular2 apps yet, we've just built demo customer calculator app for our in-house evaluation. Wonderful comparison page on Vue.js website states that Angular2 is a good framework which share a lot of concepts with Vue.

Vue.js

In short, Vue.js is a thing that I've been waiting for a long time ( I will be talking about Vue.js 2 which got quite a few improvements over first version of Vue and this is the current stable framework version). For me, in terms of elegance and conciseness, and focus on getting things done, Vue.js is the biggest change to JS after the day when I was blown out by jQuery in 2007.
If you look to Vue.js popularity graphs you will notice it is not just me: https://www.google.ru/trends/explore?q=vue.js,react.js,angular.js
Vue.js is one of the most rapidly growing JS frameworks in 2016, and I think it's not just another hype based on fans that switch to newer JS framework every 3 months, or authority (and money) of one big company.

Laravel added Vue.js to core, which is a big thing.

Pros of Vue.js

Vue.js hits a sweet spot between readability&maintainability  and fun. A spot between React and Angular 1, and if you look at Vue guideline, you will instantly notice how many nice things it got from these frameworks.
From React, it got component-based approach, props, one-way data flow for components hierarchy, performance, virtual rendering ability, and understanding of importance of proper state management of apps.
From Angular, it got similar templates with good syntax, and two-way binding when you need it (inside single component).

Vue.js is very easy to start - I've seen this in our team. It does not enforce any compilers by default, so it's really easy to drop in Vue to your legacy codebase and start improving your jQuery mess with good JS code.

Right amount of Magic

Vue.js is very easy to work with, both in HTML and JS - you can do pretty complex templates without losing your focus on business task and the template usually maintains great readability even when it gets really big - at this moment you've usually made a good progress in terms of business task solved, and you might want to refactor templates and split them into smaller components - at this moment you see the whole "picture" of your app a lot better than at the moment when you've started.

From my experience, this differs vastly from approach that I used to have in React: I saved myself a lot of hours here. In React, you have to split components into micro-components and micro-functions at the time of writing the initial version of your code - or you will literally get buried in the mess of your code. In React you will spend a lot of time polishing the props and refactoring your super-small components (that will never be re-used later) again and again since you don't see clearly if you have to change the flow of your app logic somewhere in the middle of the code writing process.

Working with html forms is a breeze in Vue. This is where two-way binding shines. It does not bring any issues to me even in complex cases, though watchers may remind of Angular 1 at first glance. One-way flow with callback passing is always at your service when you do your components splitting.

If you want some compiler magic, linting, PostCSS and ES6 - you got it. Vue extension seems to become a default way of writing public components in Vue.js 2. By the way, idea of scoped CSS of component, working out of the box, is something that looks really nice and can reduce need in proper css hierarchy naming and technologies like BEM.

Vue.js has pretty simple and useful state and props management in core, via data() and props() methods - they work great in real world. Better separation of concerns available via Vuex  (which is to my understanding similar to Mobx in React - with some mutation of state involved).

I think a good percent of Vue.js use cases won't ever require such a state management as Vuex provides, but it is always good to have an option.

Cons of VueJS

1. The biggest one: not descriptive runtime errors in templates. This is pretty similar to Angular 1. Vue.js manages to give a lot of useful warnings for your JS code, for example there are warnings when you try to mutate props, or using data() method incorrectly - the good influence of React can be very well seen here. But runtime errors in templates are still a weak point of Vue - exception stacktraces in a lot of times are not useful and are leading into Vue.js internal methods.
2. The framework is young. No stable community components - a lot of them were built for Vue.js 1, and it is sometimes not easy for newcomers to see from github repo which version of Vue the library is built for. This issue is leveled by the fact that you can do huge things in Vue without any additional libraries - you will probably just need some ajax library (vue-resource will be a good choice if you don't care about isomorphic apps, axios otherwise), and probably vue-router which is considered as core library with good support.
3. Chinese comments in code across most of community libraries - this is not surprising, Vue.js is getting very popular in China (the author speaks Chinese).
4. Single-guy project? Not exactly a real issue, but something to consider. Evan You is the guy who built Vue, after working at Google and Meteor. Laravel also used to be a single-guy project, it is still a huge success, but you never know..

Vue.js in Drupal

Disclaimer: we do not plan to use Drupal 8 any time soon at Qwintry, since we are switching to faster and simpler PHP&Node.js frameworks, and our legacy codebase is Drupal 7.
Since our legacy system qwintry.com is powered by Drupal, it was very important for us to test this new framework here, in the wild. I am not proud by a lot of code in our legacy codebase, but it works and generates our revenue, so we respect it, improve it, and build a lot of new features here. Here are the list of things I've built in Vue&Drupal already:
In-place node editing for complex order entities. This includes generating invoices for customers, and quick edit of product items. It required building basic JSON api for loading and saving of nodes - nothing too fancy, just a few menu callbacks.
Two REST-powered dashboards for proprietary Saas software we are using so our customer support does not have to login to separate websites to quickly check information related to specific customer - everything is built right inside customer profile in our website now.

I know a lot of backend developers are still stuck in 2010 and Drupal 7 core Ajax system.
I know how complex Drupal might be when you try to build some fancy multi-step ajax interaction form using core features - it's just crazy hard to maintain this code later. Yes, I am looking at you, ctools_wizard_multistep_form() and you, ajax_render!
At the same time, these Drupal developers are pushed forward by modern world UI requirements, but they might be scared by increased complexity of modern JS frameworks. Yep, it's me a year ago. Let me tell you - you won't find a better time and way to improve your interfaces than get Vue.js now, put it in your /sites/all/libraries, add it via drupal_add_js() to your template, and start hacking around. You will be shocked how easier it is to maintain a number of pure JSON callbacks sitting in your hook_menu  when the client side, including forms, is completely powered by Vue.

Vue.js in Yii2

Fun fact: Yii was created by a Chinese speaking guy - Qiang Xue. So, you might call Yii+Vue stack not just very difficult to pronounce, but also a Chinese stack :)
For our new version of Qwintry.com (not yet public) we chose Yii2 which I believe is one of the best and fastest PHP frameworks available. It is definitely not as popular as Laravel which is rocking the PHP world now, but we are pretty happy with our Yii2 stack now (though we are looking at Laravel from time to time, the guys are doing a great job).
We are gradually reducing amount  of html generated by Yii2 and PHP, and concentrating more on REST backend which generates JSON for our client-side which is powered by Vue.js. It is mostly API-first approach for all our Active Record models.
Here, we are taking API seriously and that's why we are spending a lot of time building good API documentation even though it is only used in-house.
With PHP7&latest MySQL the response times of our Yii2 JSON backend does not differ much from Node.js backends (15-20ms are the numbers I am talking about), so it's more than enough for our needs, and 10-20 times faster than we could imagine to achieve using Drupal. At the same time, it's good&old PHP with all the power of composer libraries and stable codebase at our hands.
So, Yii2&Vue.js responsiveness is huge, and in terms of code it's a pleasure to work with.

We are also using Vue.js in a number of internal projects.

Conclusion

We've been writing Vue.js code every day for around 3 months in various projects, with impressive results. 3 months is nothing in backend world, but it is something in JS world :) We'll see how it goes further.

I expect Vue to become a primary JS framework in 16-24 months if Evan You makes right steps, at least around backenders and smaller teams of frontenders. I still consider React stack to be the primary JS framework of 2017, especially if React Native manages to mature and improve itself with same pace it used to.

UPD: this post got to frontpage of HackerNews and there is useful discussion with 200+ comments there: https://news.ycombinator.com/item?id=13151317
It also got to top posts of Reddit webdev, 70+ comments here:  https://www.reddit.com/r/webdev/comments/5ho71i/why_we_chose_vuejs_over_react/

For the final code scroll down to the end of the post, since now I will be showing some ugly code that you don't really need :)

These links are created in the node--[type].tpl.php in my theme:

<a href="<?= url('admin/rotate/' . $node->field_photo[LANGUAGE_NONE][0]['fid'] . '/cw/field_photo') ?>">&#8635; Rotate CW</a>  
<a href="<?= url('admin/rotate/' . $node->field_photo[LANGUAGE_NONE][0]['fid'] . '/ccw/field_photo') ?>">&#8634; Rotate CCW</a>

the first version of the function:

<?php  
function bdr_warehouse_rotate_image($fid, $direction, $field_name) {  
  $file = file_load($fid);

  $img = image_load($file->uri);
  image_rotate($img, $direction == 'cw' ? 90 : -90);
  image_path_flush($file->uri);
  $result = image_save($img);
  if ($result) {
    $nid = db_query("SELECT entity_id FROM {field_data_{$field_name}} WHERE {$field_name}_fid=:fid", array(':fid' => $fid))->fetchField();


    db_query("UPDATE {file_managed} SET filesize=:size WHERE fid=:fid", array(':size' => $img->info['file_size'], ':fid' => $fid));
    db_query("UPDATE {field_data_{$field_name}} SET {$field_name}_width=:width, {$field_name}_height=:height WHERE {$field_name}_fid=:fid LIMIT 1", 
      array(':width' => $img->info['width'], ':height' => $img->info['height'], ':fid' => $fid));

    db_query("UPDATE {field_revision_{$field_name}} SET {$field_name}_width=:width, {$field_name}_height=:height WHERE {$field_name}_fid=:fid LIMIT 1", 
      array(':width' => $img->info['width'], ':height' => $img->info['height'], ':fid' => $fid));


    cache_clear_all("field:node:$nid", 'cache_field');
    drupal_set_message('Image rotated! Use ctrl+f5 if the image preview is not rotated - it is your browser cache.');
  }

  if (!empty($_SERVER['HTTP_REFERER'])) {
    $mark = strpos($_SERVER['HTTP_REFERER'], '?') === false ? '?' : '&';
    drupal_goto($_SERVER['HTTP_REFERER'] . $mark . 'refresh=1');
  }

}
?>

Take note that we flush image presets cache, node field cache, and also update the sizes of the picture in three places - in filemanaged table, in fielddata{fieldname}  table, and in fieldrevision{field_name} table.

But it turned out that it is not enough so I had to come up with some way to force chrome to reload picture, so we pass ?refresh=1 to the url and we had this in our init hook:

<?php  
function bdr_warehouse_init() {  
  if (!empty($_REQUEST['refresh'])) {
    drupal_add_js(drupal_get_path('module', 'bdr_warehouse') . '/scripts/img_refresh.js');
  }

}
?>  

and the contents of the javascript file were like:

function updateQueryStringParameter(uri, key, value) {  
  var re = new RegExp("([?&])" + key + "=.*?(&|$)", "i");
  var separator = uri.indexOf('?') !== -1 ? "&" : "?";
  if (uri.match(re)) {
    return uri.replace(re, '$1' + key + "=" + value + '$2');
  }
  else {
    return uri + separator + key + "=" + value;
  }
}

setTimeout(function() {  
    // force image refresh in browsers after rotation
  var x = document.querySelectorAll(".field-type-image a, .image-widget-data a");

  var i;
  for (i = 0; i < x.length; i++) {
    x[i].href = updateQueryStringParameter(x[i].href, 'rand', new Date().getTime());
  }  


  var x = document.querySelectorAll(".field-type-image img, .image-widget img");
  var i;
  for (i = 0; i < x.length; i++) {
    x[i].src = updateQueryStringParameter(x[i].src, 'rand', new Date().getTime());
  }

}, 300);

I had to enclose the image refresh function to setTimeout because Drupal was throwing 503 errors if the image was reloaded too quickly (it is the image.module locking mechanism at work). BTW, don't use this updateQueryStringParameter since awesome jquery.bbq is included in drupal 7 core :)  (but you dont need this js piece of code at all in the final version so read further)

I also realized that I want to show the rotate buttons in the node edit, in the  image file widget, as well.

Here comes the second version of the function which renames the image file by adding some symbols at the end of the file (and it forces chrome to reload the original file and all the image presets, a lot more reliably than the javascript refreshing mechanism above). Finally, it works! And we can get rid of all javascript mentioned above.

Here is the

Final code:

the main function:

<?php  
   function bdr_rotate_image($fid, $direction, $field_name) {
  $file = file_load($fid);

  $img = image_load($file->uri);
  image_rotate($img, $direction == 'cw' ? 90 : -90);
  $result = image_save($img);
  if ($result) {
    $uri = $file->uri;
    $ext = substr($uri, -3); // Change this if you expect some weird extensions like .jpeg !
    $new_uri = substr($uri, 0, -4) . '_1' . '.' . $ext;
    file_move($file, $new_uri);
   // it is not completely ok to pass php variables into sql just like that, but we do it here since the input is always safe in my case
    $nid = db_query("SELECT entity_id FROM {field_data_{$field_name}} WHERE {$field_name}_fid=:fid", array(':fid' => $fid))->fetchField();


    db_query("UPDATE {file_managed} SET filesize=:size WHERE fid=:fid", array(':size' => $img->info['file_size'], ':fid' => $fid));
    db_query("UPDATE {field_data_{$field_name}} SET {$field_name}_width=:width, {$field_name}_height=:height WHERE {$field_name}_fid=:fid LIMIT 1", 
      array(':width' => $img->info['width'], ':height' => $img->info['height'], ':fid' => $fid));
    cache_clear_all("field:node:$nid", 'cache_field');
    drupal_set_message('Image rotated!');
  }

  if (!empty($_SERVER['HTTP_REFERER'])) {
    drupal_goto($_SERVER['HTTP_REFERER']);
  }

}
?>

the node edit field widget override with rotation links (put in your theme template.php):

<?php

function YOURTHEMENAME_image_widget($variables) {  
  $element = $variables['element'];
  $output = '';
  $output .= '<div class="image-widget form-managed-file clearfix">';

  if (isset($element['preview'])) {
    $output .= '<div class="image-preview">';
    $output .= drupal_render($element['preview']);
    if (user_access('create page content')) {
        $output .= "<br> <a href='" . url('admin/rotate/' . $element['#file']->fid . '/cw/' . $element['#field_name']) . "'>&#8635; Rotate CW</a>";
        $output .= "&nbsp;&nbsp; <a href='" . url('admin/rotate/' . $element['#file']->fid . '/ccw/' . $element['#field_name']) . "'>Rotate CCW &#8634;</a>";
    }
    $output .= '</div>';
  }

  $output .= '<div class="image-widget-data">';
  if ($element['fid']['#value'] != 0) {
    $element['filename']['#markup'] .= ' <span class="file-size">(' . format_size($element['#file']->filesize) . ')</span> ';
  }
  $output .= drupal_render_children($element);
  $output .= '</div>';
  $output .= '</div>';

  return $output;
}

?>

The registered path in YOURMODULE_menu hook:

<?php

$items['admin/rotate'] = array(
    'title' => 'Rotate',
    'page callback' => 'bdr_rotate_image',
    'page arguments' => array(2, 3, 4),
    'access arguments' => array('create page content'),
  );
?>

Qwintry Logistics (I will use QWL further in text to avoid typing these words again) is an IT system which provides our b2b customers (US stores and freight forwarders) a new way for a high-quality and affordable delivery to Russia, with simple API for integration. We are not directly competing with monsters like USPS or UPS for cross-border delivery, but we are doing very similar thing here - delivery from stores/warehouses to customer door (or pickup point - which in many ways can be more convenient for the end customer than courier delivery to door).

Under the hood the system connects Qwintry warehouses (pickup hubs, if we use right terminology) in Oregon and Delaware, multiple US companies for trucking freights to airports, IATA agents and companies that book air freight for us, customs brokers (to do customs clearance in Russia), and multiple delivery companies in Russia/Belarus/Kazakhstan. All these multiple partners are connected into single workflow, and we get beautiful tracking for each individual package as a result, without investing billions of dollars to build our own infrastructure (that's where we differ from UPS and USPS):

At the same time, the IT system and some know-hows and experience in automation of fulfillment and logistics, that we got while working on Drupal-powered Qwintry b2c website (freight-forwarding business with 100k+ registered customers)  allow us to get the amazing quality and speed while keeping costs low and pricing matrix attractive.

Framework selection: Drupal8, Symfony2, Yii2

When we decided to start building the QWL website, our team had plenty of experience in Drupal 5/6/7 and some dirty hands in Drupal8, and decent experience in Symfony2 (several big projects on air), with my mind still being restless due to the fact that none of these frameworks completely met following requirements for the future QWL system:

1. being blazing fast for authorized users and API calls
2. configuration in code, no configuration in db
3. ability to write less code, with the code being beautiful, clean, and fun to write
4. being simple PHP platform to quickly hire developers without skyrocketing budgets

Why not Symfony2?

While a lot of members of our team was (and still are) in love with sf2 - I was not, since writing sf2 code was never a fun for me - never even remotely looking like fun. Yes, I'm lazy, and I'm probably not this perfect developer which sf2 is built for :)
I realized that the level of sf2 complexity and over-engineering is too big for me - very clearly - when I noticed that I kept forgetting what was the business (real) task I was trying to solve - just after five minutes of staring into a bunch of routing or configuration files in our not-too-complex project, or into Factory to generate some basic form. I just dived into the code and tried to solve issues that sf2 was throwing in my face most of time. Not business issues, but issues related to level of perfect abstraction Symfony2 provides me. I think that's a fundamental difference between me and sf2 - is that I want to solve real business issues quickly without diving too much into this game of absolute extend-ability and clean dependency-injectionability this great and 100% testable framework provides :)
My opinion can change some day, but right now I believe that most projects around me and processes around me are not that complex, but they still have tight deadlines and the quicker I can provide real value to the business with my code - the better.

Why not Drupal?

At the same time all the problems of Drupal -

• big db structure, where writing some raw db queries is a pain - you do ten joins to get basic data from 10 fields - (don't even try to explain that to financial analytics who are great in what they do, but have very basic SQL knowledge and read-only access to db),
• bad performance for authenticated users and very bad performance in node_load/node_save,
• ugly code you need to write (well, comparing to Wordpress it's a great code, but lets face it - it is not good comparing to modern frameworks with good ORM/AR),
• configuration in db

• were still there, and we were very well aware about all these issues since Qwintry.com is a big project with 30+ custom modules created by our team - starting from Endicia/USPS integration and ending with our own coupons engine, full-fledged order picking system UI, and highly customized referral program - it's all there, and we've been maintaining it for several years with impressive number of incoming packages and orders, so every line of code that could fail - failed at least several times :)

In big projects during relatively long maintenance periods you quickly realize that this Drupal code has issues:

<?php  
$order->field_paid[LANGUAGE_NONE][0]['value'] = 1;
node_save($order);  
qwintry_log($order->nid, 'Order was paid');  
?>

Because sometimes node_save with single changed field still fail due to innodb locks or whatever, and you at least need to wrap it into try { } catch blocks like this:

<?php  
$order->field_paid[LANGUAGE_NONE][0]['value'] = 1;
try {  
  node_save($order);
  qwintry_log($order->nid, 'Order was paid');
} catch (Exception $e) {
  drupal_set_message('Critical warning for operator');
  qwintry_log($order->nid, 'Exception thrown during invoicing');
}

?>

Why am I posting this example? Because when your projects grow that big so you have everyday problems like that (and you write a lot of custom code for new features, so your custom code base size is comparable to framework size) - it may be a good sign that you probably need some lower level framework (lower than Drupal), where transactions are there for all objects, working mostly automatically, and where it is a lot cheaper in terms of cpu and speed to save objects to db in 'traditional' way of framework - so you don't do ugly dbinsert or dbquery("INSERT INTO {table}") instead of node_save when you need to process just 30-40 objects at a time.

I was even considering plain PHP, but it would be stupid in 2014 to invent own wheel :)
Another idea was to use Laravel, and I'm pretty sure it would be a viable option - never heard bad things about it - not too complex, but still elegant and fast.
But, instead of that, we decided to try Yii2 which was in alpha quality by the time when we started - and it turned out to be a great framework.

Why Yii2?

Historically, Yii is the most popular PHP framework in Russia, and probably in China (no links to chinese IT websites, sorry) - since the author of the framework, Qiang Xue, is Chinese (lives in US). It is not hugely popular in US, as far as I know.

Big popularity is not a sign that the framework is good (or bad) - but it usually means you can find developers easier and faster. Finding good Drupal developers is a big issue. Finding sf2 developers is easier, but still can be a problem. Finding Yii developers is easier than finding sf2 developers :) You have to be careful while testing them since the learning curve is not that big in Yii - so there can be quite a number of bad programmers among these guys - those PHP guys that  Ruby and Python devs make jokes of  :)

I have a lot of friends in Russia who have been using Yii1 for years, but during these years I've been successfully doing Drupal projects and we honestly never had a lot of projects where custom code base can be bigger than framework+contributed modules code base, and good speed under highload was not a big factor, so for that type of projects it would be stupid (and too expensive) to use Yii.
I still think that for those kind of projects (where you don't write big - and 2-3 small custom modules+customized theme is not big - amounts of custom code) Drupal is a perfect engine. You basically develop most of stuff at the speed of prototyping, you have access to huge amount of great contributed modules and to drush - it's just amazing how quick the complex web development can be nowadays.
But now, when I look back and think of projects that we've implemented in Drupal and which had huge amount of development hours invested in it (thousands of hours on code writing) - I think that Drupal was not the perfect solution for these - it gives more issues than value.

So, I used Yii1 in my sandboxes for some time. Than, when we finally needed some PHP framework for serious code writing - we started to use Symfony2 (read my blog post about this selection here) - the Yii1 already felt outdated, while Yii2 was not even remotely ready for usage. And the Symfony2 was exciting to try, giving us the new modern tools like Composer and Twig.

The Yii core devs decided to rewrite second version of their framework almost from scratch, just like Fabien did with Symfony2. It was a long-lasting project, and there was a risk to loose all the community that will switch to fresh sf2, Laravel and Phalcon and never come back. But now I think it was worth it (and I don’t think Fabien regret his sf2 rewrite, as well). The beta of Yii2 was released in April of 2014 (and it was a raw beta). We started working with Yii2 and QWL project in March of 2014 so we had several funny issues related to alpha quality of code - but all of them were resolved with later releases.

So, in QWL, we needed great performance, and the code was mostly custom (a bunch of external APIs to integrate; a huge bunch of docs to generate from db objects - mostly for customs brokers; and a need for good own API) - so it was obvious that it's not where Drupal shines.
And it was a great moment to try Yii2 since it was close to production quality already.
New kid on the block, with all the goodness of modern PHP development: composer, namespaces, autoloading, great ActiveRecord, prepared for REST APIs building. All that was implemented by slim and very simple framework core, and it works with a great real speed.

What is a real speed? I use that term to distinguish between speed of cached page performance and speed of page performance without cache.
In Symfony2 (which is not fast at all, comparing to Yii) you can't even disable all the levels of caching, so I had a number of issues with that - when your routing annotations are cached, when your Yaml configuration is cached, when your Doctrine objects and schemas are cached, and when your Twig templates are cached, and when your DI LazyServiceMapPass is probably cached - and you can't disable all these cache layers completely in development environment due to the complexity of framework features - it's easy to get lost when debugging something, and if you delete all the cache - the framework will be surprisingly slow while warming up all that.

In Yii2 you can disable all caching, and the dev environment with all the debug panels will still be fast! You can (and you should) cache everything in production to be super-fast, but it's really amazing how fast it is in development mode. It was one of the reasons to choose this framework.
Another reasons:

1) Configuration of components is dead simple, and is written in PHP:

if you see something like

<?php  
'sms' => [  
     'class' => 'app\components\sms\SMSC',
     'user' => 'username',
     'key' => 'xxx'
 ],
?>

in website configuration file - you know that you can access this object through Yii::$app->sms and it's implementation is a class like this:

namespace \app\components\sms;

class SMSC extends Component {  
  public $user;
  public $key;
  /* some methods here */
}

so configuration params are transformed into class properties by default, and you are not required to write code for additional processing to transform configuration directives to something the class can use. Yes, it can be restrictive, and Symfony2 developers can give a bunch of examples where such configuration approach is lacking flexibility and may be it is not exactly self-documenting, but in most cases that's what you need - in most cases you don't want to know a heck about DI service compiler passes, and you don't want to create additional XML with service definition (which will be cached, remember? good luck with typos in these xmls) - you just want to create this component and start using it right away.

2) Most of decisions are made by core developers

The Yii2 core feels monolithic compared to sf2 (so core developers made own Logger, own ActiveRecord, and great RBAC permission system is also in core) - but it felt good when I was doing real work, comparing to alternatives with bigger amount of features (using Doctrine+ACL).
I like the fact that Bootstrap3 is there in core of Yii2 and there are widgets to do less typing (e.g. \yii\bootstrap\Modal for modals). I have had experience injecting Bootstrap3 into Drupal7 and I can't say it's a big pleasure to do.

At the same time, the culture of contribution in Yii is weak comparing to Drupal world.
You don't use a lot of contributed modules when you build Yii project - you mostly rely on the core and your own code. Of course, there are a lot of contributed modules but their number is ridiculously low comparing to number of Drupal user-contributed modules - at least, that was my impression. The level of extendability is significantly lower in Yii2 comparing to Drupal, but it didn't bug me a lot since I know how complex and slow infinite extendability can get. Yii2 is still in many ways better extendable than Drupal - for example, it's very easy to use your own user and session processing, and customize the registration forms - and this flexibility is more important in big projects than a way to inject something in theme hooks or change menu routing details via hookmenualter (Drupal shines here).

Now, in Yii2, most of contributed modules are hosted on GitHub (and Packagist - for composer).

3) ActiveRecord is a lot less typing than ORM like Doctrine.

Active Record is a concept that somehow mixes the object representing each table row and a super-object that can be used to retrieve specified table rows - in single class. Sorry for this terrible explanation, but here is a simple example:

Drupal 7

$shipment = node_load(db_query("SELECT entity_id FROM {field_data_field_tracking} WHERE field_tracking_value=:tracking", [':tracking' => $tracking])->fetchField()); // yes, I know about db_select, but using it in a bit more complex cases looks even uglier than that 
$shipment->field_is_sent[LANGUAGE_NONE][0]['value'] = 1;
node_save($shipment);  

Yii2

$shipment = \app\models\Shipment::findOne(['tracking' => $tracking]);
$shipment->is_sent = 1;
$shipment->save();

Symfony2

$em = $this->getDoctrine()->getManager(); // if you're in controller
$shipment = $em->getRepository('PixeljetsShipmentBundle:Shipment')->findOneBy(['tracking' => $tracking]);
$shipment->setIsSent(1);
$em->flush();

In ORM (Doctrine) the super-object whose job is to fetch rows from db is EntityRepository, while each db row is represented by Entity object, it means you need separate class for Repository and separate class for Entity, so in the code above the hierarchy of ORM is:
ShipmentRepository -> Shipment

In yii2 ActiveRecord this super-object concept is mixed into ActiveRecord class which also holds event handlers related to each-db-row objects and stuff like that.
Obviously, Active Record is created by developers who are lazy and hate typing and creating a bunch of classes, while ORM is more academic, loosely coupled, supporting SOLID, and beautiful concept, which works better for enterprise development.
But the resulting code of retrieving objects (and this is the code that you write ten times per day) of ActiveRecord is more elegant and short.

The underlying code of ActiveRecord is probably less than 10% of Doctrine code size and you can understand how it works in one day. Then try to explore the Doctrine code some day (I only did a quick overview, and yes, no surprise it's big and complex)

Some beautiful ideas of Yii2 AR I think were borrowed from Eloquent (Laravel AR), for example the relations:

<?php

class Shipment extends ActiveRecord {  
  function getAuthor() {
    return $this->hasOne(User::className(), ['id' => 'user_id']);
  }
}
?>

now you can get owner of shipment:

<?php  
$shipment = Shipment::findOne($id);
echo $shipment->author->username; // separate lazy sql query against users table executed here  
?>

or, you can do it via join:

<?php  
$shipment = Shipment::find()->where(['id' => $id])->joinWith('author')->one(); // select * from shipment left join user ... 
echo $shipment->author->username; // no sql query here!  
?>

Isn't it beautiful and expressive?
While I always had problems remembering the arguments and names of Drupal zoo of db functions (in D7 it got even worse with EntityFieldQuery and db_select - I don’t really want to repeat myself so read more about this in my previous post) - it really lacked simplicity, straight-forwardness and "it works for everything in the system" impression, - and while I didn't like Doctrine DQL and had bad experience extending this DQL syntax to add some condition to my search query - Yii2 simplicity was like a miracle for me.
You can fallback to raw sql in conditions when you need, still getting AR objects as a result:

<?php  
// get 20 shipments which are older than 1 month or which author email is test@test.com 
$q = Shipment::find()->joinWith('author')->where(['author.email' => 'test@test.com']);
$q->orWhere('shipment.create_time < :time', [':time' => time() - 60*60*24*30]);
$shipments = $q->limit(20)->all();

foreach ($shipments as $shipment) {  
  echo $shipment->tracking;
}

?>

By the way, Yii2 supports transactions out of the box, and you can save objects reliably and be sure that you don't leave your db in a bad state.

Other things to note in Yii2:

Bower

• jQuery is in core but it is not glued into the code - it is downloaded as a Bower package, and the idea of Bower manager support via Composer looks very nice (minor issue: it seems to slow down the process of composer update process). It means that Yii2 modules can specify dependencies to bower packages. (Bower is a package manager for frontend libraries, mostly for javascript libraries - built by Twitter and seems to gain some momentum now)

DB is primary, no schema generation from code

• DB is primary, while the code is secondary. Very important difference comparing to Symfony2 ORM and Drupal schema: if you remove your database completely, in Symfony2 you can get full schema of db from your ORM models, in Drupal you can do the similar thing, in Yii you can't do that easily - the AR code 'learns' about your db structure on the fly.  You need some good db admin software to do db design - but I think that SQLyog interface is superior to writing Drupal schemas by hand or using Symfony2 console utility. The Doctrine2 ability to generate ALTER queries from changed models still looks very cool :)  In Yii2, to do migrations, you go to db admin, alter the table using its ui, and copy&paste the resulting ALTER to migration file, that is later pushed to git. In db there is a separate table to remember migration states. This approach is pretty simple and gets things done.

Template layer

• the core template engine is plain PHP - the only thing I don't like here is that it's easy to forget to escape some user data, and it's too much typing to escape everything:

<?= \yii\helpers\Html::encode($var) ?>  

I don't like this, so I made a shortcut function:

function e($var) {  
  return \yii\helpers\Html::encode($var);
}

and now in templates I use

<?= e($var) ?>  

• still not as beautiful as {{var}} in Twig, and not OOP, but I can live with it (for now).

BTW, there is a Twig integration for Yii2 - maintained by Yii core devs. Not tried it yet.

The big difference between Drupal templating and Yii2 templating is that Yii2 templates mean more typing and copy&pasting when you start - since you need to add real template (view) for each form, and for each page, and for each model CRUD, while in Drupal when you create new node type or simple form you don’t need to take care of creating any template, you need to think of it only if your need some fancy html structure.
At the same time, I found it easier to manage html structure of complex form in Yii2 than in Drupal7 (which has a great theme layer - but you can write a book on how you do specific things here - and that is a killer for newbies).

<div class="row">  
        <div class="col-sm-3">        
        <?= $form->field($model, 'broker_status')->dropDownList(Shipment::getAvailableBrokerStatuses(), ['prompt' => '']) ?>
        </div>
        <div class="col-sm-4">        
        <?= $form->field($model, 'broker_message')->textarea() ?>
        </div>
 </div>

to do this basic styling in Drupal you have to fight with formalter (add #fieldprefix to mimic the required structure) or override template completely which means you need to add separate theme callback - and while it is not a rocket science - lazy developers may feel a bit upset when designer says that he needs this element floating there  and he can’t do it just via CSS, using default Drupal form html structure.
In Yii2 you go this extra-mile at the start, using Gii which generates CRUD for models from db table, and then you have precise control over the html, it is easy to modify and support - and all the the templates in the system work the same way - and you know exactly in which file some html structure of registration form located. In Drupal, during maintenance period of some complex project, you always need some time to figure out how this specific div was generated - was it some form_alter from contributed module adding it, or was it your div from your custom theme?

In Symfony2, the Twig engine is great, and I can’t say anything bad about it - it is similar to Yii2 approach. I’m afraid the Twig won’t save Drupal8 theme layer from complexity unless devs are ready for big functionality degradation (because Drupal flexibility is complex, and if you convert all these theme_ callbacks to .twig files - you will also get performance degradation that will slow down Drupal even further - and cache cleanup will be a bigger pain than it is now.. but we’ll see the result when Drupal8 is released)

Forms

In Yii2 forms are usually created on the base of some model.
That is a nice concept - ActiveRecord and form models have validation rules described in same syntax - with rules() and scenarios() methods inside model class. Note that the amount of typing is minimal, and you can pass anonymous function for custom validation rule - and it’s again no redundant typing, and quick result.

Client side validation is automatically generated from validation rules described in model class.

The nice ‘side effect’ of same validation rules used not just for forms, but for base Model class is that you can launch your validation for some ActiveRecord object simply by calling

<?php  
if (!$shipment->validate()) {  
$errors = $shipment->getErrors(); 
}
?>

Menu routing

When you need to create another route in Drupal7, you need to implement hook_menu() and append your new path to the registry of routes:

<?php  
function devel_generate_menu() {  
  $items = array();

  // Admin user pages
  $items['admin/generate'] = array(
    'title' => 'Generate items',
    'description' => 'Populate your database with dummy items.',
    'position' => 'left',
    'page callback' => 'system_admin_menu_block_page',
    'access arguments' => array('administer site configuration'),
    'file' => 'system.admin.inc',
    'file path' => drupal_get_path('module', 'system'),
  );
  return $items;
}

?>

in Symfony2 you do the similar thing but there are plain routes there, no such metadata like menu title. So, you need to create xml file like this:

<?xml version="1.0" encoding="UTF-8" ?>

<routes xmlns="http://symfony.com/schema/routing"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://symfony.com/schema/routing http://symfony.com/schema/routing/routing-1.0.xsd">

    <route id="fos_user_registration_register" pattern="/">
        <default key="_controller">FOSUserBundle:Registration:register</default>
    </route>
</routes>  

or, you can deal with annotations and specify routes via annotations in controller:

<?php  
class PageController extends Controller  
{
    /**
     * @Route("/pages/{id}.{_format}", name="pages_view"})
     */
    public function viewAction($id)
    {
    // action code here
    }
}
?>

I loved the idea of annotations at the first sight, but in practice, in development it turned out to be a pain due to caching issues (when something specified in annotations do not work, you have to cleanup cache every time you change your annotation).
The fact that you can specify routes with different ways across the same project is also frustrating for me.

By the way, in Drupal8 the way to specify menu routes is pretty much the same as #1 way in Symfony2 - via [modulename].routing.yml file.

and now the typical Yii2 routing:

<?php  
class PageController extends Controller  
{
   public function actionView($id) {
   // action code here
   }
}
?>

that's right - you don't need to write anything to get the route into the routes registry, it's just the convention that action* methods in controller are turned into routes. In this specific case the ?id=xxx query parameter is also required and system throws 403 error if it is not there. Of course, you need to add access callbacks to restrict access but you can do it later when you need it, via separate conventional methods in the controller - it is done by attaching behaviors.
In development you don't even need to cleanup cache so new action starts working!
This whole idea that routes are not described separately can be shocking, but it works pretty well and it's a breeze when you add a lot of ajax callbacks with separate paths.

...

You also get  CLI interface (which is easy to extend by putting controller in special folder), access control, various logger types (logging to files, db, syslog, email with granular setup is available), basic web ui for rapid html and model generation (called gii), mongodb support, memcache and redis support for caching (file caching is default), sphinx support.

Implementation of specific features:

Some notes regarding API authentication: in Yii2 this part is pretty well thought through and there is a separate handbook page for this. In QWL, when customer uses the web panel - he is logged in through sessions and cookies, and during the api call the current user object is created on different authenticator mechanism - header tokens in our case - and this didn’t require a lot of developer time to implement.

Another interesting task was creating a dashboard where operator can take a quick look at the packages that require special attention (if they are sitting for too long in pickup point, or they have active issue flag, or some status in tracking is incorrect and unexpected, etc). That is a very important feature that allows us to monitor the health status for each part of logistics chain.

We have pretty big and advanced dashboard for operators on Qwintry b2c website (Drupal powered), and it is implemented  using Panels and Views. It has 15+ panes with misc stats and to make it open in adequate time that human can wait (2 seconds) we had to move most of panes to ajax loader (that was a custom hack for Panels) - otherwise the loading times could be 10+ seconds.

We also implemented flexible file storage for documents - when package is created - all the files are put to local filesystem for faster access and preview, and when the shipment is marked as “delivered” - all the files are moved to S3 for long term storage, to save local file space.
We’ve created File AR model with ‘storage’ field and when the ‘storage’ field is changed to s3 - files are copied to s3 - during afterSave hook of the model. We’ve used AWS sdk for php for this, and very thin Yii2 component so s3 is available as a service.

We’ve also integrated littlesms.ru api for SMS notifications - and it was also available as a service throughout the system so sms are sent like this:

<?php  
Yii::$app->sms->sendMessage($phone, $message)  
?>
and in project config it is:  
<?php  
'sms' => [  
            'class' => 'app\components\sms\LittleSMS',
            'user' => 'user111',
            'key' => 'pw'
        ],
?>

when littlesms.ru was discontinued we switched to SMSC api, but we didn’t change any code using the sms service, we just created new (very simple) component on the base of SMSC code and changed the system config:

<?php  
'sms' => [  
            'class' => 'app\components\sms\SMSC',
            'user' => 'userSMSX',
            'key' => 'pw2'
        ],
?>

both components had same method sendMessage (it could be the same Interface with sendMessage method or even an AbstractClass that both classes inherit) so the change was seamless for the remaining code.

Another cool feature of QWL is implementation of AMQP protocol for queues of tasks (unfortunately, still nothing like that is implemented in core).
For example, when we need to generate documents for broker, it can be an archive with 200+ pdf documents.
We use mpdf for pdf generation, and it is not quick at all, unfortunately. So it’s obvious that this is  a long running task that needs to be executed asynchronously.

We use this task queue when sending sms and email messages, generating pdf documents, generating document archives, and in other places.

The QWL also has a registry of courier companies and a set of methods so final delivery company is selected on the base of multiple parameters (pickup point selected by customer, address city, box size, box weight, etc) - but that's a topic for separate post. Pickup points map contains merged locations from multiple courier companies.

Conclusion on framework selection

Yii2 is simple, and if you have good PHP background - it will be easy to learn.
When I was learning it, and looking into how things are done here, I got this great feeling that authors of Yii2 did a lot of stuff exactly the same way I would do it in a "perfect framework" that I would create.
Many times my thought was “These guys are lazy just like me” :) this great balance of simplicity and flexibility, the balance between having fun writing code and still good code readability in long term maintenance is hard to achieve.
I’m pretty sure some developers would say that Yii2 is a bit quick&dirty, and not perfectly testable (by the way, there is a Codeception integration in core of Yii2 development package - so it is definitely testable!), and services and configuration management are too simple here, and static calls to get services are ugly - most of it is true, especially comparing to sf2 - and I’m ok with that opinion - it’s just the matter of personal preference, experience, targets, number of developers in the project, and project type. I try to think about business targets most of time when we write our code. I saw lots of great talented developers that dream about 100% test coverage and spend hours discussing which PHP DI component is better - and then they fail meeting the deadlines or just distort the real task so it does not comply to customer needs, but fits into perfect architecture of system instead.
At the same time, nobody wants to get ugly unmaintainable code in repo, me neither.

It’s all about balance :) and for me, Yii2 seems to have good balance.
QWL project is in production, it’s used by customers, and it’s a pleasure to maintain and extend it. We'll see how it goes.
So, for our team, it was a period of ~7 years of mostly Drupal development (since 2005), and then ~2 years when our team was building Drupal7 projects and Symfony2 projects, and now in 2014-2015 we have Drupal, Symfony2 and Yii2 projects to maintain. Luckily, bigger framework selection in our case does not mean increased complexity of development - it means quite the opposite - the tools now fit the tasks even better and we see better what is the right framework for the future project.

1. Not-too-great experience with high load projects, powered by Drupal 7

Okay, Drupal is good enough until you get a project with big expectations in terms of response time for authenticated users. Nginx proxy, boost and memcached help a lot with anonymous page hits, but things are sometimes not good enough when we talk about authenticated user saving some nodes (1 field = 2 inserts to fields and revisions table, 40 fields - and you get super-slow node saving, at least on non-ssd hosting), using ajax stuff like dynamic forms (Drupal form ajax is not bad, it's just not super-blazing-fast).
Things are getting even worse if you have big website with lots of views and fields, and you just clear the cache on production - your server can become close to non-responsive.
That's what happened with us in several projects.
Okay, you can avoid using views and write your own replacement for fields which will store data in single table, and own ajax that do not pass all page ids of the page back to the server, - and it will probably be almost as fast as lower level framework code (yii, symfony, whatever) but what's the point in using Drupal then? The code will be uglier (let's face it, Drupal API is not as clean and decoupled as Symfony2 API). The amount of time spent on the development will probably be close.

2. Settings in DB

Drupal was built to allow anyone to create complex website without writing a line of code.
It sounds like an unrealistic dream for most of website developers, who are using raw PHP/Ruby/Python/whatever, but not for developers who are using Drupal! Latest version of Drupal and Views is indeed very close to this amazing dream.
As a side effect, when you are creating project where you need lots of programming, you are struggling with settings-in-db approach.
Of course, there are Features which allow to do an impressive deployment stuff.
But then, when you get a bug in features during deployment, which occasionally deletes fields from user profile on production (we indeed encountered this bug in production once - god bless db backups!), you think "I'd better write my own code, it's too much magic".
Still, I think that Features are a brilliant piece of functionality and approach.

3. One of our biggest customers fall in love with Symfony2

These guys asked if we'll be able to create new projects using some robust framework, suggesting Symfony2 after making their own research. It was a matter of big projects, thousands and thousands of man hours. Why not?

4. Drupal8 is using Symfony2 components

That was another sign that Symfony is a good choice.
Just like we've chosen jQuery or git for all our projects because of the fact that Drupal decided to go with them, and not with mootools, prototype, and mercurial with bazaar. For business needs, it's almost always good to use mainstream technologies, though it's not as geeky and cool as writing a CMS in node.js.

5. Our team (and me, mostly) was eager to learn new system after 7 years of (mostly) Drupal development

Oh yeah. The grass is always greener on the other side of the fence, you know :)
While Drupal is moving forward, some stuff in PHP world was still relatively new for Drupal in 2012. (Annotations, composer, twig, ...)
And it's interesting to use this stuff in real project, not on localhost!

That's it. Now it's more than a year passed, and you know what?
After two months of work with Symfony (we were building big and complex ecommerce solution), I realized that it will be a big mistake to switch over completely.

Drupal is still a very good choice for 85% of websites in the world. You don't need high load and ultra flexibility in 85% (probably, 95%?) of world websites.
And Drupal can be a better choice than Symfony2 for all these projects.
The main (the only?) reason is - rapid and easy development in Drupal.

You can create working Drupal website with the speed of prototyping (if you are good Drupal developer with good experience).
Okay, all the hardcore features still require coding, but you can really deliver  working stuff in hours, not days of work.
Symfony is good, but it's a lot more manual work. The good sign of the fact that this problem is real - is the existence of Symfony RAD edition bundle.

I have a good example - right now when we hiring Symfony2 developer, we ask them to build basic website which contains just registration (basic stuff, FOSuserbundle, we ask to remove username field, just leave email and password) + profile (firstname, lastname) + Facebook login (HWIOAuthBundle).

The average time of implementing this by developers is ~7 hours. And most of the time is spent in writing and testing pretty big yaml configurations for bundles.
The same stuff in Drupal will take ~2-3 hours - drupal install, email_registration module, and https://drupal.org/project/hybridauth or similar for facebook login.
So, it's harder to do "rocket launch" in Symfony.

Some things that are available in Drupal out-of-the-box (like, permission system) are not easy to do in Symfony2. Typically most of sf2 projects contain pretty ugly checks against current user role. And it's a bad practice comparing to permissions checking - if you use role checking and you need to add another role which can do the same action, you will have to modify code to add this new role.
Of course, you can implement some kind of security voter or ACL in sf2, and it can be smarter and more flexible than in Drupal, but nobody wants to mess with it without REAL reason. And that's what I call poor engineering - when "right" choices are too painful so most of developers tend to stick to "wrong" choices.

Symfony components

Symfony components (Doctrine ORM, HttpKernel, Forms, Routing, Doctrine annotation parser, ...) leave good impression - they are completely decoupled, and it is really a way to go in terms of architecture, not surprising that Drupal8 uses a lot of Symfony components.

At the same time some third-party bundles leave impression that their creators are in love with over-engineering, not in getting things done.

ORM and db queries

Drupal 7 dev team made a lot of efforts to build own ORM-like code for entities (EntityFieldQuery) but Doctrine2 ORM and query builder are far more robust and polished, and play nicely with Symfony models and annotations, this part is really beautiful in sf2 - that's a no brainer that Symfony is stronger than Drupal in this part.
Writing your own db queries in code is a pleasure in Symfony, and not a big pleasure in Drupal 7, especially if you are writing 'raw' SQL - the main reason is that Drupal 7 fields are a victim of own flexibility (they allow you to change single-value field to multiple-value on the fly, and they are also designed for multilingual websites)  - and it requires separate table for each field, while in Symfony2 you have to decide if the field is single-value (than it's a field in a model table) or multiple-value when designing your db structure. Multiple-value field is typically not a field, but a separate model in sf2.
The Drupal 7 approach is good when you use it from admin interface, but not that good when you dig into code, you need three joins against tables with ugly long names like "fielddatafieldsendinvoice_paid" just to build a condition to check three fields of a node (unless you use EntityFieldQuery, which designed only for loading nodes - so it's really a performance hit to use it, that's why it's not too popular in our code).

Third-party infrastructure

Whole third-party infrastructure of sf2 relies on Github, Composer and Packagist heavily, and it leads to difficulties in tracking which version of the package is good for your installation - for example, some packages can break when you go from Symfony2.1 to Symfony 2.2.
If you still develop in production (ouch!), I bet you will spend a lot of hours trying to understand what to write in composer.json to return your website back to life.
Of course, it happens mostly because Composer and Github were created to make happy everyone in PHP world, not for specific framework.
But for beginner, Drupal module system with 6.x/7.x branches of all modules and themes is way easier to understand.

Another annoying thing is that deployment to production in Symfony2 is significantly more complex than in Drupal.
In Drupal, you commit all third-party modules to single git repo of your project, then you launch git pull, drush updatedb, drush cc on production and you are (typically) good.

In Symfony2, all vendors (third-party modules) are not stored in central git of your project - since most of them have own git repos.
So, when you deploy to production, you need to launch composer install and wait while all packages will update (and your production will be offline during this time).
And if some package git repo is offline - you are in trouble.

Of course, there is a right way to do the deployment - Capifony, based on ruby-powered Capistrano. But you need time and efforts to get used to it.

Code maintenance

But when you're done with complex project in Symfony2, I would say it's way easier to maintain it over time, and the code is cleaner and easier to understand by another developer. Though, for me, it's still not a big joy to write lots of these OOP classes when you are developing stuff. You need time to get used to it.
The fresh example of what I don't like in sf2:
If you want to output standard translated message "The item was created" to flash (flash is session-stored message, the equivalent of drupalsetmessage() in Drupal) you need to put:

$this->get('translator')->trans('flash.create.success', array(), 'JordiLlonchCrudGeneratorBundle');
Good luck trying to type it in without editor shortcuts.

and if you compare it to

t('The node was created successfully');

you probably get the idea of the difference between Symfony and Drupal which frustrates Drupal developers :)
Yes, that's a global function t() - it's not super architecturally correct comparing to translator service in Symfony, which you can easily override when you need, but.. I don't care about overriding translator in every project, but I still use standard translator in every project, and I usually use crud generator, and it's really a lot of typing!
Every time when I write this, I wonder, if this nice guy, JordiLlonch, creator of CrudGenerator bundle, was thinking about all the developers who will have troubles remembering his name, especially 'iLl' part? :)
Of course, you can create your own version of global t() (or extend base controller and add YourController::t function) in Symfony2.
But
1) you will have to do it in every project
2) you will have to explain to your team that there is a non-standard shortcut
3) (in case you do it via global function) you will have this bad feeling that you are going against architecture of your framework :)

Performance

Symfony is definitely not about "we have lean and mean code which is fast".
It's about "we have good, a bit bulky, but good-for-everyone architecture and 5 levels of caching, and then we have another level that caches cache of cache".
Let's just say that single Doctrine ORM has 3 types of cache.
And you can't completely disable caching of everything, even in dev environment.
The only reliable way to test configuration/annotation changes is to delete cache every time you modify it.

Symfony2 is significantly faster than Drupal. It implements lazy-loading and autoloading everywhere and it's nice - memory footprint is relatively small. You can update the memory limit - instead of 128mb (drupal) you can (typically) use 64mb (sf2) for the same functionality.
Doctrine is also relatively fast in production. It is clever enough to make single big update/insert for several objects when you call flush(), instead of ugly multiple node_save which can take lots of time to complete in Drupal.

Conclusion

Symfony2 is more "academic", OOP, professional and serious, faster in terms of performance, easier to maintain for a team of developers.
Drupal is more fun, less code to write, and faster development and deployment, sometimes quirky, quick, and dirty. It quickly moves to OOP in it's parts, while other parts are still procedural. That's what I can call 'dirty', too.

The same project will cost roughly twice as much developed in Symfony, comparing to Drupal, and will take twice as much time.
I think the learning curve is approximately the same for Drupal7 and Symfony2.

So, we're still doing Drupal development now.
At the same time, we were lucky to hire pretty talented sf2 developers, so we now do a lot of Symfony2 projects (5 complex projects so far) and even our Drupal team start enjoying it - but these are still two separate teams, one doing Drupal, one doing Symfony2. And I'm pretty sure that knowledge of Symfony2 will help us when we start doing Drupal8 development - that's what I call synergy :)

upd: due to good search engine positions spammers seem to love this post, so comments are now disabled.

Query:
SELECT transaction_id, amount FROM transactions WHERE amount>20.5

Response:
123|20.5

Query:
SELECT transaction_id, amount FROM transactions  WHERE amount=20.5

Response:
No rows

So, 20.5 is really greater than 20.5 in sql database. Sometimes.

Oh yeah, it looks like balance_tracker developers were not aware of that (until today). So, it's like a bomb with timer for hundreds of websites which rely on this and other modules which use floats to store monetary-like amounts in db and are doing some sql selects which compare values stored in db against something. In our case, it was 20.563333333... stored in balance, which prevented our auto-payments system (built on top of balance_tracker+ubercart) in our big project from working properly.

How does it happen?

When you calculate some value in PHP (and you use floats in PHP, of course):

<?php  
 $total = 3 / 2; // total is now 1.5555555555... in PHP
?>

and store it in sql table in cell of type float, it is stored as 1.55555555555..
(but it shows as 1.5 when you are looking at the value using phpMyAdmin or SqlYog - which is understandable, but really adds confusion!)

That's why you need decimals. When you put 1.55555555.. in cell of type decimal, it is stored as 1.55. So, it works as most usual people expect it to work when dealing with financial stuff.
And when you use Drupal fields to store monetary amounts in your nodes, use decimal as field type, too!

UPD: another good (best, I guess - but requiring more work) solution (as suggested in comments) would be to use integer column in mysql and operate with integers in PHP/SQL, only rendering values as floats to website users.

My plan was to provide rules event "The package [tracking number] changed active state from [old state] to [new state]". (words in square brackets are Rules arguments).
On Qwintry website, "Package" is a node, with "tracking number" textfield.

So, basically, my Rule was:

if [tracking number] changed active state from [old state] to [new state]:  
   - fetch entity by property "tracking number" = [tracking number]   
   - send [author of loaded entity] a nice email about state change

Easy, huh?

I've implemented _rules_event_info() hook in my module code, and created my Rule.
The rule worked perfectly when I triggered the event using my admin account, but..

.. it didn't work when I run website cron to trigger my event.

event was triggered but it couldn't find the node with such property (though I knew that I have node with such tracking number in db).

After hours of debugging I found out that the issue here is that "Fetch entity by property" action of Rules uses EntityFieldQuery, which of course respects node access permissions and checks current user access. My "package" nodes were private for their owners. And cron uses anonymous user to trigger the event! So, cron didn't have enough permissions to load the nodes.

That makes perfect sense (for lots of use cases!) to check node access unconditionally, but in my case it was a big trouble.
I think that the perfect solution for the issue would be an "ignore access permissions" checkbox in "fetch entity by property" Rules action.

I've created an issue in Rules issue queue: http://drupal.org/node/1804586 but didn't get any replies yet.

I hope this will save some time to someone else!