JMX Scraper: YAML file and integration test hadoop #1675
Conversation
JMX Scraper - YAML config and integration test for Hadoop
| export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS" | ||
| export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -Dcom.sun.management.jmxremote.authenticate=false" | ||
| export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -Dcom.sun.management.jmxremote.ssl=false" | ||
| export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999" |
There was a problem hiding this comment.
[minor] would it be possible to avoid having this script file by just setting the HADOOP_NAMENODE_OPTS with appropriate JMX settings directly ?
There was a problem hiding this comment.
What about all those other env vars too tho?
There was a problem hiding this comment.
As discussed with @robsunday yesterday, this looks to be a modified copy of a shell script of the hadoop distribution, which might have been modified to configure the JMX configuration.
The current startup script might allow to easily provide this configuration, but quite often it's not possible to easily override through a single env variable, and sed surgery isn't the easiest thing to maintain.
There was a problem hiding this comment.
So I tried few ideas how to get rid of this custom script, but without a luck. All tips/guides I've found on the internet recommend modification of hadoop-env.sh (as it is done in our test).
At first glance it looks like passing your own env var HADOOP_NAMENODE_OPTS to docker should work.
Unfortunately, Hadoop launches multiple processes during startup and executes this file many times to setup env variables and the ossue is that initial content of HADOOP_NAMENODE_OPTS passed to docker is somehow lost.
Only hardcoding JMX stuff in hadoop-env.sh worked for me, but if you know about reasonable alternate solution I'll be happy to use it.
There was a problem hiding this comment.
Thanks for trying to remove it, we should be fine to leave it as-is since we don't have a better option.
Co-authored-by: Magda Wojtowicz <[email protected]>
Scope
Part of #1362