This Week in Elasticsearch - July 30, 2014
Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
EC2! DDoS! Cloud! Elasticsearch!
If you've been reading the news, you've been seeing those words together quite a bit for the past few days. Don't panic! (And bring your towel.) We've got you covered.
tl;dr:
- Block ports 9200 and 9300 from all machines not part of your development environment
- Don't run Elasticsearch as root
- Disable dynamic scripting for non-sandboxed environments, which is disabled by default in our 1.2.x and later branches. If you would like to use dynamic scripting, make sure you set up your environment securely by following these guidelines.
- Keep current on the latest version of Elasticsearch as we're constantly making improvements to our software, including updating our security defaults. You can download the latest version here, and there are instructions on how to upgrade here.
Elasticsearch core
- Fielddata: Fix the ordinals impl for sparse fields (#6908, master and 1.4)
- Fielddata: remove comparators (#5980, master and 1.4)
- DocValues: Change numeric data types to use
SORTED_NUMERIC
docvalues type instead of a custom encoding inBINARY
(#6967, master and 1.4) - Search & Count API: Add option to early terminate doc collection (#6876, master and 1.4)
- Aggregations: change to default
shard_size
in terms aggregation (#6857, master and 1.4) - Internal: streamline use of IndexClosedException when executing operation on closed indices
(#6988, master and 1.4)
- Aggregations: Make
_source
parsing intop_hits
aggregation and search api consistent (#6997, master, 1.4 and 1.3) - Sorting: Speed up string sort with custom missing value (#7005, master and 1.4)
- Aggregations: Improve JSON output scoping (#7004, master, 1.4 and 1.3)
- Internal: expose the indices names every action relates to if applicable (#6933, master and 1.4)
- Filter cache: remove use of recycled set in filters eviction (#7012, master and 1.4)
- Aggregations: Added pre and post offset to histogram aggregation (#6605, master and 1.4)
- Flush API: Add
wait_if_ongoing
option to _flush requests (#6996, master and 1.4) - Geo: Fix parse error with complex shapes (#5773, master and 1.4)
- Core: Use the provided cluster state instead of fetching a new cluster state from cluster service (#7013, master and 1.4)
- Caching: Add a periodic cleanup thread for
IndexFieldCache
caches (#7010, master and 1.4) - CORS: Support regular expressions for origin to match against (#6923, master and 1.4)
- Aggregations: Added an option to show the upper bound of the error for the terms aggregation (#6696, master and 1.4)
- Mapping: Add
multi_field
support for MapperexternalValue
(#6867, master and 1.4) - Aggregations: The
nested
aggregator should also resolve and use theparentFilter
of the closestreverse_nested
aggregator (#7048, master, 1.4 and 1.3) - Aggregations: fixed value count so it can be used in terms order (#7050, master and 1.4)
- Circuit breaker: Add
HierarchyCircuitBreakerService
(#6739, master and 1.4) - Internal: Support parsing lucene minor version strings (#7055, master, 1.4 and 1.3)
- Internal: use
AtomicInteger
instead ofvolatile
int for the current action filter position (#7021, master and 1.4) - Rest: fixed filters execution order to be from lowest to highest rather than the other way around (#7019, master and 1.4)
- Transport Client: fixed the node retry mechanism which could fail without trying all the connected nodes (#6829, master and 1.4)
- Transport Client: Don't add listed nodes to connected nodes list in sniff mode (#7067, master and 1.4)
logstash is just a way to get people hooked on using ElasticSearch for everything, I think. Well played, @jordansissel
— Michael Pearson (@mipearson) July 26, 2014
Elasticsearch Ecosystem
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
- Elasticsearch 1.3.1 has been released. Note that this release includes an important bugfix, so make sure to read the release notes!
- There have been a whole bunch of plugin releases: elasticsearch-transport-memcached, elasticsearch-transport-wares, elasticsearch-mapper-attachments, elasticsearch-lang-javascript, elasticsearch-lang-python, elasticsearch-analysis-phonetic, elasticsearch-analysis-smartcn, elasticsearch-analysis-kuromoji, elasticsearch-analysis-stempel, elasticsearch-analysis-icu and elasticsearch-transport-thrift
- Jettro Coenradie authored this great walk through of the Top Hits & Scripting features in Elasticsearch 1.3.x.
- Florian Hopf has continued his excellent article series on Elasticsearch Use Cases, this time focusing on it as a flexible query cache.
- Sergey Khaladzinski teaches you how to use Elasticsearch to build a faceted navigation system. He's even got some love for aggregations in there, too!
- Joseph Jude shared this lovely how to on using the ELK stack to analyze Twitter data, which also includes a link to a nifty how to on setting up the ELK stack.
- Security aficionados, Victor Julien has all the news you can use on using the ELK stack to monitor log flows from Suricata.
- We were named one of the 10 hottest big data start ups of 2014. So far. Awww, shucks.
Slides & Videos
Learn how Elasticsearch powers the Building Performance Database, helping folks understand their building's energy footprint
Florian Hopf shared his slides on Elasticsearch Use Cases, as presented at Java Forum Stuttgart 2014.
Rotem Hermon, the organizer of the Elasticsearch Tel Aviv meetup, on serendip.me How Clairvoyant uses the ELK stack for log analysisWhere to find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Community Manager, Leslie Hawthorn, know!
Germany
- Elasticsearch is a proud sponsor of eurucamp 2014. If you'll be attending, make sure to say hello to Colin Surprenant and ask him about our open positions on the Logstash engineering team.
- If you're heading to FrOSCON on August 23rd & 24th, we're happy to bring you bouncy castle love this year. And make sure to see Isabel Drost-Fromm on Your Search Doesn't Work.
Israel
The OpsTalk Meetup Group will get together on August 4th for a full day workshop on the ELK stack. Register now to save your place!
Poland
The Warsaw Java Users Group is holding a Microservices Hackathon on August 9th from 10 AM to 10 PM. You can spend those twelve hours hacking on the ELK stack and a variety of other tools. Register now to attend.
Minneapolis people, Elasticsearch is kicking off a meet up in your city: http://t.co/xQ47AC1eW9
— Michael Ducy (@mfdii) July 29, 2014
We hope to see you on Thursday!
United States
- If you haven't gotten enough of the Pacific Northwest with all things OSCON, then head on over to CascadiaJS on July 31st - August 1st. Joe Fleming from the Kibana engineering team will be attending, so say hello to him in the hallways!
- Heading to Chicago for LinuxCon and Cloud Open North America? Say hello to Leslie Hawthorn, and check out her panel Empowering Your Corporate Open Source Software Developers.
Where to Find You
Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, take a moment to let her know.
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.
Training
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- New York - August 12, 2014 (core Elasticsearch training)
- Sao Paulo - August 14, 2014 (core Elasticsearch training)
- Rio de Janeiro - August 18, 2014 (core Elasticsearch training)
- Amsterdam - August 20, 2014 (core Elasticsearch training)
- Boston - August 27, 2014 (core Elasticsearch training)
- New York - August 27, 2014 (ELK workshop)
- San Francisco - August 28, 2014 (ELK workshop)