Rsyslog, EC2 an Hostnames

We are automating our server farm using amazon’s ec2. Part of this is collecting our log files using http://www.papertrailapp.com (like loggly, etc). Unfortunately, using rsyslog, we’re not seeing the system names show up properly.

to reproduce:

  • we create an AMI of a well operating server with updated code, etc..
  • that server has the hostname ec2-123-123-123-13
  • we have it configured to launch and get up and running
  • as expected, every server gets its newest hostname
  • rsyslog initiates, and starts sending log data to papertrail
  • the server name passed in the rsyslog events is the original c2-123-123-123-13
    (for example, the two lines below, 5 minutes apart, show the original system name and the new system name as the rest of the log.

” Jun 30 00:45:11 ec2-54-147-195-63 system: ec2-54-161-201-58.compute-1.amazonaws.com
Jun 30 00:50:11 ec2-54-147-195-63 system: ec2-54-161-201-58.compute-1.amazonaws.com

  • this is incredibly sticky. i’ve tried to add restarts within rc.local of both apache and rsyslog.
  • i can go into the box directly and restart apache and syslog, and it will tend to reset to the correct server name and start streaming.

unfortunately, this means that all our logging happens at the level of the staging server that we use pre-production. it also makes it very hard to debug, since all the servers look the same.

interesting observations:
– when logging in (ssh) with username ubuntu, the prompt is still the OLD name
– when sudo bash to log in as root, the prompt is the new name
– when logging in much later / a second time, the prompt is the new name
– we thought this might have something to do with EIPs and specific servers. unfortunately, even when we created a 3rd generation server, the initial (and sticky) IP address was that of the immediately preceding server.
– i’ve tried to schedule a cron job @boot to reset rsyslog, or in the rc.local, but to know avail. it seems just to get stuck further.

rsyslogd 5.8.6.
ubuntu

any suggestions, help? how can we reset the name and effectively use our remote logging?


Source: ubuntu

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.