BotProxy: Rotating Proxies Made for professionals. Really fast connection. Built-in IP rotation. Fresh IPs every day.
I am currently migrating our existing ELK application from EC2 to Amazon Elasticsearch Service. Firstly, I decided to keep my existing Kibana server and only switch it to point to the new ES domain. So I had changed my existing kibana configuration, as advised in this section: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-kibana
Then, I started to have some issues, which most of them were addressed in this article: https://stackoverflow.com/questions/33150013/kibana-wont-connect-to-elasticsearch-on-amazons-elasticsearch-service Except for step 4. Kibana was still requiring authentication via the header. Since I am not using CLI or a AWS SDK to authenticate via access keys, but using HTTP/S calls instead (i.e. http://search-[es-domain]:80 inside the kibana config file), it is looking like I would need to sign my HTTP requests anyway (as suggested in step 4 on the previous link) however, I am looking for other options to overcome this issue, in order to have my own Kibana server, cluster, with the proper level of access control from the outside internet, whilst still protecting access to the Elasticsearch for only certain IAM users/roles.
I then decided to try another option: to implement my own proxy (in this case use my existing Nginx server to make it point to the new ES domain. This approach is suggested in slide 56 on the Reinvent bdt209-launch-amazon-elasticsearch-for-realtime-data-analytics slideshare. That way, I could make the web server accessible to the world (via ports 443/80), with of course basic web authentication, whilst protecting the ES Domain with very restrictive access policy only allowing IP-based (as well as IAM roles) access to the Elasticsearch cluster.
However I still got into the same issue. I'm getting this message as a reponse:
"message": "'a2liYW5hYWRtaW46YWRtaW5AMTIz' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic a2liYW5hYWRtaW46YWRtaW5AMTIz'."
Which means I would need to sign the request. Can I please have some suggestions on how to overcome this issue? Do I really need to sign the web requests programatically? Or are there other options without affecting security and access control (meaning allowing restricted public access to kibana, with very limited access (role and IP-based) to ES cluster) ?
Thanks very much.
The root of the issue was Nginx having added the Authorization header (enabled via Basic Authentication module) that includes an encrypted string with the login details to my custom Kibana instance.
If this header is passed to the ES domain endpoint, then ES domain will try to use that header to authenticate against the receiving server. Since it is not preset for my custom authorization credentials, it will simply reject with the Authorization error.
So I had to disable Header Forwarding from the NGINX proxy to the ElasticSearch endpoint replacing my location stanza (in my NGINX config) with this one:
location / {
proxy_pass http://localhost:5601/;
proxy_pass_request_headers off;
proxy_redirect http://localhost:5601/ /;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/nginx.auth;
}