Hello world!

Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!

Posted in Uncategorized | 1 Comment

Linux EtherChannel – Double Your Pleasure Double Your Fun

As always I’m long overdue for an update. I’ve been busy with other more important things, but I figured since I had to document this for others within my group, here would be the best place since it might benefit others as well.
 
For today’s exercise we’ll be creating an EtherChannel between a Red Hat Enterprise 5 server, and a Cisco Catalyst 3750 Switch. This is actually far simpler then it sounds, and can be completed in about ten minutes.
 
We’ll begin with configuring the IEEE 802.3ad Dynamic link aggregation (AKA EtherChannel) on the Red Hat Enterprise Linux server. Begin by logging in via SSH, Telnet or directly on the console itself. I do recommend having access to the console directly, so should anything go wrong and you lose network connectivity you’ll be able to easily change things back.
 
Once logged into the server, switch user to "root" if you’re not already logged in as root. Change directory to "/etc" and modify the "modprobe.conf" file using your favorite text editor such as "vi". I personally like using "nano". Add the lines in bold from the example "modprobe.conf" below to your file. Then save your changes and return to the bash prompt.
 
Sample /etc/modprobe.conf
alias scsi_hostadapter megaraid_sas
alias scsi_hostadapter1 usb-storage
alias eth0 bnx2
alias eth1 bnx2
alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1
Next we need to create a network script for the "bond0" interface that we defined above in the "modprobe.conf" file. This will be used to configure the network properties for the virtual adapter. Once again, use your favorite text editor to create a new file called "ifcfg-bond0" in the "/etc/sysconfig/network-scripts" directory. In this file you will define the device name used above"bond0", IP address, gateway, network mask etc for the virtual adapter. Below is an example. 
 
Sample /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.0.0
NETMASK=255.255.255.0
IPADDR=192.168.0.25
USERCTL=no
GATEWAY=192.168.0.1
TYPE=Ethernet
IPV6INIT=no
PEERDNS=yes
 When you’re done configuring the properties of the virtual adapter, save your changes and exit the editor.
 
The next step is to modify the network script for each adapter that will be added to the EtherChannel. The adapters that we’ll be using in this server are eth0 and eth1. Please note your interfaces may be different, so check before continuing.
 
Start by modifying "ifcfg-<int>" using your text editor, where <int> is the interface name. In this case my file name is "ifcfg-eth0". Add the proper references to the virtual adapter created above "bond0" and remove any IP information such as IP address, gateway, netmask etc since that information will be handled by the virtual adapter. Below is an example of the "ifcfg-eth0" file. Note the bold items are required for the EtherChannel to function properly.
 
Sample /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:11:22:33:44:55
ONBOOT=yes
MASTER=bond0
SLAVE=yes

TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes
 Repeat the steps above for each additional interface you add to the Etherchannel.
 
Sample /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=66:77:88:99:aa:bb
ONBOOT=yes
MASTER=bond0
SLAVE=yes

BOOTPROTO=none
TYPE=Ethernet
USERCTL=no
Now that each physical adapter has been associated to the virtual adapter, it’s now time to create the EtherChannel on the Cisco switch that this server is connected to. In this example I’ll be using two Cisco Catalyst 3750G switches in a stack. In addition to the added benefit of increased bandwidth that the EtherChannel provides, having multiple switches will add switch level redundancy to the EtherChannel.
 
Begin by logging into the switch, entering configuration mode, and creating the EtherChannel interface. The EtherChannel interface, also known as a "Port-channel", is a virtual interface. You can use any number from 1-48 that isn’t already assigned to another EtherChannel. In this example I’ll be using "28". Configure all the properties such as VLAN, switchport mode etc. that you would normally define on the physical interfaces. Please note that the properties defined in the EtherChannel will override any interface configuration you have defined on the physical interfaces this Etherchannel is assigned to.
Switch# config terminal
Switch(config)# interface Port-channel28
Switch(config-if)# description SERVER
Switch(config-if)# switchport access vlan 123
Switch(config-if)# switchport mode access
Switch(config-if)# spanning-tree portfast
Next we’ll need to configure the physical interfaces by associating them to the EtherChannel we created above. I like to mirror the configuration of the EtherChannel on the physical interfaces, so in the event the EtherChannel on the far end of the link becomes misconfigured the server will still be accessible. The line in bold is the only required change that needs to be made on the physical interfaces.
Switch(config)# interface GigabitEthernet1/0/8
Switch(config-if)# description SERVER-1
Switch(config-if)# switchport access vlan 123
Switch(config-if)# switchport mode access
Switch(config-if)# channel-group 28 mode passive
Switch(config-if)# spanning-tree portfast
Repeat the process for each additional interface in the EtherChannel.
Switch(config)# interface GigabitEthernet2/0/8
Switch(config-if)# description SERVER-2
Switch(config-if)# switchport access vlan 123
Switch(config-if)# switchport mode access
Switch(config-if)# channel-group 28 mode passive
Switch(config-if)# spanning-tree portfast
 It’s now time to restart the network service on the Linux server. Optionally you can reboot the Linux server if that’s easier.
[root@SERVER network-scripts]# service network restart
Shutting down interface eth0:  /etc/sysconfig/network-scripts/ifdown-eth: [  OK  ]
Shutting down interface eth1:  /etc/sysconfig/network-scripts/ifdown-eth: [  OK  ]
Shutting down loopback interface:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface bond0:  [  OK  ]
Check the status of the EtherChannel on the Linux server. If the EtherChannel is operating properly you should see "Bonding Mode: IEEE 802.3ad Dynamic link aggregation" when you concatenate "/proc/net/bonding/bond0" 
[root@SERVER ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: fast
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 28
        Partner Mac Address: 01:23:45:67:89:ab
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:11:22:33:44:55
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 66:77:88:99:aa:bb
Aggregator ID: 1
 To check the status of the etherchannel on the Cisco switch, use the "show ether-channel" command.
Switch# show etherchannel port-channel
 
Group: 28
———-
                Port-channels in the group:
                —————————
Port-channel: Po28    (Primary Aggregator)
————
Age of the Port-channel   = 12d:18h:33m:47s
Logical slot/port   = 10/28          Number of ports = 2
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Port security       = Disabled
Ports in the Port-channel:
Index   Load   Port     EC state        No of bits
——+——+——+——————+———–
  0     00     Gi1/0/8  Passive            0
  0     00     Gi2/0/8  Passive            0
         
Time since last port bundled:    7d:07h:50m:28s    Gi2/0/8
Time since last port Un-bundled: 7d:07h:51m:31s    Gi1/0/8
At this point you should have a fully functional EtherChannel. If you have any questions feel free to post comments, or contact me directly.

Posted in Computers and Internet | 1 Comment

Scheduling Device on/off with an APC MasterSwitch

This week I had need to control the on/off status (power) of a device in my home on a regular schedule, but also wanted the flexibility of overriding the status remotely via the Internet. Sure this sounds like a perfect application for X10 devices, but being a network geek I have parts lying around my home that are equally capable of performing the same job. Even better, they’re free, since I already own.
 
I dragged out an old APC MasterSwitch model AP2111 that I had lying around the house. The MasterSwitch allows individual control of each receptacle in the PDU via HTTP, Telnet, and SNMP. Unfortunately the MasterSwitch does’t have any built-in ability to schedule power state changes (on/off). Current MasterSwitch models can be fairly expensive, but used, discontinued models like my AP2111 can be had for less than $50.00 on eBay.
 
Since the MasterSwitch supports SNMP we can leverage the free tool Net-SNMP to remotely change power state of the receptacles. NET-SNMP is available on most flavors of Unix such as Linux and Solaris. NET-SNMP is also available for Windows.
 
When used in conjunction with a powerful program scheduler such as cron (unix) or Windows Scheduled Tasks, we can fully automate the powering on or off of devices connected to the MasterSwitch on a regular schedule. Here’s a quick how-to.
Assumes the MasterSwitch is connected to the network, has and IP address and is accessible via HTTP and SNMP from another computer.
 
1) Enable SNMP on the APC MasterSwitch
1.1) Login to the APC MasterSwitch and expand "Network" from the left hand navigation.
1.2) Click "SNMP"

 

1.3) Under the heading SNMP select "Enabled" and click "Apply"

2) Configure the SNMP Write Community string on the MasterSwitch

2.1 Under "Access Control" enter a SNMP community string that will be used to access the MasterSwitch. "Private" is the default.
2.2 In the "NMS/IP" field enter the IP address of the machine that will be accessing the MasterSwitch, or use "0.0.0.0" for any.
2.3 Ensure the "Access Type" is set to "Write"

2.4 Click "Apply"

3) Using a Windows machine that already has NET-SNMP installed, create a new scheduled task
3.1 Click ‘Start" – "Control Panel" – "Scheduled Tasks" – "Add Scheduled Task"
3.2 When the wizard begins click "Next"
 
3.3 Click "Browse" and navigate to the NET-SNMP BIN directory (Default is "c:usrbin") and select "snmpset.exe"

   

3.4 Name the the task and select the frequency that it will run.

3.5 Enter the account credentials that the process will execute under. Note: This user must have read/execute permission to "snmpset.exe"

 

3.6 Check the box "Open Advanced propertied for this task when I click Finish" and then click "Finish"

3.7 In the task "Run" field, add the necessary parameters specifying the MasterSwitch IP address, community string, outlet number and action in the following format.

"C:usrbinsnmpset.exe" -v 1 -c {community} {ipaddress} 1.3.6.1.4.1.318.1.1.6.5.1.1.5.1.1.{outlet} i {action}

{community} is the SNMP community string specified earlier in step #2.3
{ipaddress} is the IP address of the APC MasterSwitch
{outlet} is the electrical outlet number on the MasterSwitch that the device you wish to control is connected
{action} is the power function you wish to perform.  See table below

(1) Immediately turn the outlet on
(2) Turn the outlet on after delay
(3) Immediately turn the outlet off
(4) Turn the outlet off after delay
(5) Immediately reboot the outlet
(8) Cancel pending command

The example below connects to the APC MasterSwitch at "10.0.0.35" using the community string "private" and immediately turns off outlet 5.

"C:usrbinsnmpset.exe" -v 1 -c private 10.0.0.35 1.3.6.1.4.1.318.1.1.6.5.1.1.5.1.1.5 i 3

 

If needed, create an additional scheduled task to power the device back on.

As always, if you have any questions post them here. Enjoy  

Posted in Computers and Internet | 19 Comments

Using Cisco WCS with Microsoft IAS

I once again find myself documenting the undocumented as I continue my crusade to eliminate independent authentication databases in favor of more centralized approaches such as LDAP and RADIUS.

I covered a similar topic before in a segment entitled APC Web/SNMP & RADIUS Authentication where I walk through integrating APC management hardware with a Microsoft IAS back end. Thus eliminating the need for independently managed local usernames/passwords on each piece of APC equipment. This also allowed administrators to use their familiar Active Directory credentials, and improved auditing and logging since changes were tied back to the individual who logged in instead of a generic "device specific" local administrator account.

This segment will cover integrating Cisco’s Wireless Control System (WCS) with Microsoft’s Internet Authentication Service (IAS) which is Microsoft’s implementation of a Remote Authentication Dial-in User Service (RADIUS) Server.

For the purposes of this LAB I’m using three servers and one workstation, all of which are located on the same subnet as pictured below.

  If you haven’t already done so you’ll need to register IAS with Active Directory.

Create a RADIUS Client

First we need to create a RADIUS Client in IAS that will represent our WCS Server. In IAS right click on "RADIUS Clients" and select "New RADIUS Client"

The new RADIUS client wizard will walk you though the process of adding the RADIUS device.

To keep things simple and avoid any potential confusion, In the "Friendly name" field I named the object "WCS-SERVER".

Next, populate the "Client address (IP or DNS)" field with the IP address or DNS name of the WCS Server and then click "Next".

On the following screen select "RADIUS Standard" from the Client-Vendor list. Then enter a shared secret (password) that WCS will use to authenticate to IAS. In this case I’m using the shared secret "WCS". Confirm your shared secret and click "Finish". Do NOT check the box "Request must contain the Message Authenticator attribute".

Note: Shared secrets are case sensitive.

Create a RADIUS Policy

Now we need to create a Remote Access Policy. We’ll begin by defining who has access to WCS. To begin right click on "Remote Access Policy" and select "New Remote Access Policy"

When the New Remote Access Policy Wizard begins click "Next".

Select "Set up a custom policy" under "How do you want to set up this policy?" Then create a name for your policy. I’ve decided to name mine "Cisco WCS". Once you’ve named your policy click "Next".

On the Policy Conditions page click "Add" and select "Windows-Groups" and click "Add" again.

In the Groups page click "Add" and select the Active Directory user group you want to have administrative access to WCS. In my case I have added the Domain Admins group. Click "OK" when you have finished adding groups.

Once again on the Policy Conditions page click the "Add" button and choose "Client-Friendly-Name" from the list and click "Add".

In the Client-Friendly-Name window type "WCS-SERVER". This will ensure that this Remote Access Policy only applies to the "WCS-SERVER" RADIUS client that we defined ealier. When you are done click "OK".

Your Policy conditions should look similar to the image below. If everything looks good then click "Next".

On the Permissions screen select "Grant remote access permission" and click "Next".

On the Profile page click "Edit Profile" and go to the "Authentication" tab. Check the box labeled "Unencrypted authentication (PAP, SPAP). Uncheck all other boxes.

Next, click on the "Encryption" tab and clear all the checkboxes except "No encryption"

It’s at this point that things begin to get interesting. Click on the "Advanced" tab and remove all attributes that are listed, and then click "Add". Select "Vendor-Specific" from the list and click "Add".

Click "Add" Again and the "Mutivalued Attribute Information" window appears. click "Add" one last time to add an attribute value to the list.

In the "Vendor-Specific Attribute Information" click on the vendor list and select "Cisco". Select the radio button "Yes it conforms" and lastly click "Change Attribute".

It is here, in the Vendor Specific Attribute (VSA) that we define the RADIUS response messages sent back to WCS upon a successful authentication.

We begin by changing the "Vendor-assigned attribute number". This number will always be "1" for all WCS VSA(s).

The first "Attribute value" that we’ll define is the WCS roll that the user who successfully authenticated to RADIUS will have in WCS. Earlier we defined that Active Directory Domain Admins would be the only users who would be able to successfully authenticate to this RADIUS policy. It’s only fitting that these same users be WCS Admins. So in the "Attribute value" we enter "Wireless-WCS:role0=Admin" and click "OK".

At this point you might think we’re done in IAS but far from it. It seems that not only must you define the roll that the authenticated user will be member of, but you must also specifically define the permissions the user will have inside WCS. At this time there are currently 45 "tasks" within WCS that can be controlled through RADIUS. You must include each task as a VSA that you want your users to have access to in WCS. For instance, I want my Domain Admins to have access to the Audit Reports within WCS. So I need to include a VSA with the appropriate Value as such.

Below is a listing of all current WCS tasks as of version 4.2.62.0 in proper VSA format. They will need to be added individually to grant  users logging in to WCS full access.

Wireless-WCS: task1 = Audit Trails
Wireless-WCS: task2 = TACACS+ Servers
Wireless-WCS: task3 = RADIUS Servers
Wireless-WCS: task4 = Logging
Wireless-WCS: task5 = Licensing
Wireless-WCS: task6 = Scheduled Tasks and Data Collection
Wireless-WCS: task7 = User Preferences
Wireless-WCS: task8 = System Settings
Wireless-WCS: task9 = Diagnostic Information
Wireless-WCS: task10 = View Alerts and Events
Wireless-WCS: task11 = Email Notification
Wireless-WCS: task12 = Delete and Clear Alerts
Wireless-WCS: task13 = Pick and Unpick Alerts
Wireless-WCS: task14 = Severity Configuration
Wireless-WCS: task15 = Configure Controllers
Wireless-WCS: task16 = Configure Templates
Wireless-WCS: task17 = Configure Config Groups
Wireless-WCS: task18 = Configure Access Points
Wireless-WCS: task19 = Configure Access Point Templates
Wireless-WCS: task20 = Configure Choke Points
Wireless-WCS: task21 = Monitor Controllers
Wireless-WCS: task22 = Monitor Access Points
Wireless-WCS: task23 = Monitor Clients
Wireless-WCS: task24 = Monitor Tags
Wireless-WCS: task25 = Monitor Security
Wireless-WCS: task26 = Monitor Chokepoints
Wireless-WCS: task27 = Access Point Reports
Wireless-WCS: task28 = Mesh Reports
Wireless-WCS: task29 = Client Reports
Wireless-WCS: task30 = Inventory Reports
Wireless-WCS: task31 = Performance Reports
Wireless-WCS: task32 = Security Reports
Wireless-WCS: task33 = Location Server Management
Wireless-WCS: task34 = View Location Notifications
Wireless-WCS: task35 = Maps Read Only
Wireless-WCS: task36 = Maps Read Write
Wireless-WCS: task37 = Client Location
Wireless-WCS: task38 = Rogue Location
Wireless-WCS: task39 = Planning Mode
Wireless-WCS: task40 = Ack and Unack Alerts
Wireless-WCS: task41 = Migration Templates
Wireless-WCS: task42 = Configure Spectrum Experts
Wireless-WCS: task43 = Monitor Spectrum Experts
Wireless-WCS: task44 = Interferers Search
Wireless-WCS: task45 = Audit Reports

When you’re done entering all the tasks you want your users to have access to as VSA’s in IAS, your Multivalued Attribute Information window should look similar to the following.

We’ve finally completed the configuration of IAS. Click "OK"on each open window to confirm your changes until you’re back at the main IAS MMC.

Configure WCS

Now that setting up IAS is behind us, it’s now time to configure WCS to use the RADIUS server as its method of authentication.

We begin by accessing the web interface of WCS and logging in as the WCS administrator. Click "Administration" – "AAA" from the top navigation bar. Then select "RADIUS" from the menu on the left. On the "Select a command" pull-down choose "Add RADIUS Server" and click "GO".

In the fields enter the information about the RADIUS server that we defined earlier in the "Create a RADIUS Client". Then click "Submit"

  Once the RADIUS servers are defined in WCS we need to enable RADIUS authentication. To do this, click on "AAA Mode" from the menu on the left. select the "RADIUS" radio button and ensure "Fallback on Local" is checked so we don’t get locked out of WCS if the RADIUS server was to become unavailable.

When your done click "OK".

From here you should logout and test your RADIUS authentication. Provided that all the steps were followed correctly, you should be able to login successfully as any user in the Domain Admins group of Active Directory.

As always, if you have any questions feel free to contact me.

 

Posted in Computers and Internet | 4 Comments

Configuring SNMP on Cisco IOS devices

It’s been a long time since my last update and I’ve received quite a few requests for an update, so I thought I would carve out some time to update the blog. I’ve been working on a few different projects lately that are worthy of posting about, but none of them are complete, so for this posting I’ll be doing something fairly straightforward that everyone can use.

Simple Network Management Protocol (SNMP) has existed for a very long time and there are many tools out there that can leverage SNMP to provide useful metrics for measuring your network or server usage and health. A short list of some of these tools include the following.

This list is by no means exhaustive. It serves as a small example of the many tools available that are capable of leveraging SNMP.

The type of information you can retrieve using SNMP is too numerous to list here, but a few examples are listed below.

    • CPU Utilization
    • Memory Utilization
    • Disk Usage
    • Network Bandwidth
    • Network Errors

Before you can take advantage of what SNMP has to offer you must first configure SNMP on the device you wish to monitor.  For the purpose of this blog entry I will be focusing on SNMP read only, AKA “SNMP GET”. I will not be spending much, if any time on SNMP write, AKA “SNMP SET”.  Though it is important for you to note that not only can you access device information remotely using SNMP, in most cases you can also configure device settings remotely using SNMP.

In the examples below I will be using a Cisco Catalyst 3750 running IOS 12.2(35)SE2.

SNMPv1/v2 with Cisco IOS

Configuring SNMP v1 and v2 in IOS is a fairly straightforward process. All that is required is set a community string (name), and to define the permissions that the community will have. For instance, the example below provides read only access “RO” using the specified community string “public”. “RO” is the most common SNMP permission type. The second example provides read/write “RW” access using the specified community string “private”. “RW” access is potentially dangerous and I don’t recommend configuring a read/write community unless you have a specific need. Even then you should consider using SNMPv3 and/or limiting SNMP access to a specific IP address.

switch(config)# snmp-server community public RO
switch(config)# snmp-server community private RW

Also, you can configure some additional SNMP options which might be helpful. When you have a large number of SNMP devices, keeping track where they are located or who is responsible for them can be helpful.

switch(config)# snmp-server location 1 Main St – Anytown USA
switch(config)# snmp-server contact Joe Smith – 800-555-1212

Lastly, if you are using a Network Management Solution (NMS) such as CiscoWorks, or SolarWinds Cirrus, that allows you to remotely upgrade code on your IOS switch, you might want to allow the NMS to reboot the switch using SNMP once the upgrade has completed successfully. To allow this you will need to configure a read/write community such as the one in the example above, as well as add the following command to your switch.

switch(config)# snmp-server system-shutdown

To test your SNMP configuration download and install NET-SNMP for your specific client operating system. I recommend downloading a build that includes SSL. This will be important later.

Once NET-SNMP is installed you can run “snmpget” to retrieve the information from your switch. The example below queries the contact information we configured earlier on the switch.

C:usrbin>snmpget -v2c -Cf -c public 10.0.32.6 system.sysContact.0

SNMPv2-MIB::sysContact.0 = STRING: Joe Smith – 800-555-1212

SNMPv3 with Cisco IOS

Configuring SNMPv3 is slightly more complicated then configuring SNMPv1 and SNMPv2, but it provides greater security by adding authentication (username/password) as well as encryption to the protocol. This is important if regulations mandate the use of high level security standards, such as Sarbanes Oxley or HIPPA.

You begin the configuration of SNMPv3 on your IOS switch by creating an SNMP “view”. A “view” defines what information the SNMP user/group will have access to when they query the device. The example below creates a view called “MyView” that includes access to the “internet”, “system”, “interfaces” and “chassis” SNMP OIDs on the device.

switch(config)# snmp-server view MyView internet included
switch(config)# snmp-server view MyView system included
switch(config)# snmp-server view MyView interfaces included
switch(config)# snmp-server view MyView chassis included

In the next step we create an SNMP group called “MyGroup”. This group is then granted access to the view “MyView” that we created in the previous step, with read only permissions.

switch(config)# snmp-server group MyGroup v3 priv write SNMPv3 MyView

Lastly, we create a SNMP user called”MyUserName” and grant it access to the group “MyGroup” that we created earlier. We will need to provide an authentication password, as well as a privacy password which will serve as the encryption key for the protocol. NET-SNMP requires that these passwords be a minimum of eight characters in length.

switch(config)# snmp-server user MyUserName MyGroup v3 auth md5 MyAuthPass priv des56 MyPrivPass

If you haven’t already done so, you’ll want to configure the SNMP location and contact information so we can test the SNMP configuration.

switch(config)# snmp-server location 1 Main St – Anytown USA
switch(config)# snmp-server contact Joe Smith – 800-555-1212

To test the SNMPv3 configuration of the switch we will once again use NET-SNMP. It’s important that you’ve installed a build of NET-SNMP that includes SSL if you plan on test SNMPv3.

Once NET-SNMP is installed you can run “snmpget” to retrieve the information from your device. When you execute the command below NET-SNMP queries the SNMP contact information we configured earlier on the switch.

C:usrbin>snmpget -v3 -u MyUserName -l authPriv -a MD5 -A MyAuthPass -x DES -X MyPrivPass 10.0.32.6 system.sysContact.0

should return the following..

SNMPv2-MIB::sysContact.0 = STRING: Joe Smith – 800-555-1212

– Good luck

Posted in Uncategorized | 59 Comments

RSPAN

I am long overdue for an update to the blog so I thought RSPAN would be something that is fun and easy to do.
 
RSPAN (Remote Switched Port Analyzer) is a method of mirroring a physical interface on a Cisco device remotely, eliminating the need to be on the same switch as the device you wish to monitor. RSPAN is a great tool for troubleshooting a server or workstation’s network communications. RPSAN can also be used in conjunction with an IDS system such as SNORT for monitoring potential threats. The best part is that this can all be done without leaving your seat.
 
Configuring RSPAN is fairly straightforward. The concept is similar to a local SPAN port, but instead of the destination being another local interface on the same switch the destination becomes a dedicated RSPAN VLAN. This RSPAN VLAN is then trunked to another switch where your capture device is located.
 
In this example I am going to RSPAN a gigabit interface on a remote Cisco Catalyst 6500 to a local Cisco Catalyst 3750 where I am located.  The Catalyst 6500 is running IOS version 12.2(18)SXF5 and the Catalyst 3750 will be running IOS version 12.2(25)SEC.

 
We start by creating a new dedicated RSPAN VLAN on the Catalyst 6500. This VLAN cannot be used for any other purpose then RSPAN.
6500#config terminal
6500(config)#vlan 199
6500(config-vlan)#name RSPAN_VLAN
6500(config-vlan)#remote-span
6500(config-if)#end
This step should be repeated on the Catalyst 3750 and any other devices between. The RSPAN VLAN will not cross layer 3 boundaries so there must be layer 2 connectivity between the RSPAN source and destination switches.
 
Then we must add the RSPAN VLAN to the trunk interface connecting these two switches
6500(config)#interface Gi2/24
6500(config-if)#switchport trunk allowed vlan add 199
Repeat this step on the Catalyst 3750 for the corresponding trunk interface
 
Now we will ensure there are no other SPAN sessions configured on either switch.
6500(config)#no monitor session all
Next, we will configure the RSPAN source on the Catalyst 6500
6500(config)#monitor session 2 source interface Gi7/1 tx
6500(config)#monitor session 2 destination remote vlan 199
6500(config)#exit
Now you should double check your work on the remote end.
6500#sh monitor session 2
Session 2
———
Type                   : Remote Source Session
Source Ports           :
    TX Only            : Gi7/1
Destination RSPAN VLAN : 199
On the Catalyst 3750 we can now configure the RSPAN destination, provided that the RSPAN VLAN was created in the steps above.
3750(config)#monitor session 2 source remote vlan 199
Again, double check your work.
3750#sh monitor session 2
Session 2
———
Type              : Remote Destination Session
Source RSPAN VLAN : 199
Destination Ports : Fa1/0/1
    Encapsulation : Native
          Ingress : Disabled
At this point you should be able to connect to Port Fa1/0/1 on the Catalyst 3750 and use your favorite capture program such as WireShark to capture traffic going to the server connected to Gi7/1 on the Catalyst 6500.

Posted in Uncategorized | 82 Comments

APC Web/SNMP & RADIUS Authentication

APC Web/SNMP cards AP9617 and newer such as (AP9618 and AP9619) with AOS firmware v2.6.4 or greater have RADIUS authentication as a feature. RADIUS allows you to use a centralized user database to authenticate against, such as active directory or local user accounts in FreeRADIUS. If you have a large number of APC devices such as SmartUPS, Masterswitches or Metered PDUs then you will probably be very interested in RADIUS to control user access.

Could you imagine creating and maintaining accounts on 100+ devices? What if you had to change the administrator password?

RADIUS is much easier to manage and maintain but it does require a little time investment up front to make the magic happen. APC provides instructions on how to configure the device to use a RADIUS server in the manual. They even include some examples on how to configure FreeRADIUS. The trouble is, I’m not running FreeRADIUS. I’m running IAS. APC also provides some very basic guidelines that should apply to any RADIUS server, but unfortunately these instructions are about as clear as mud. Frustrated with the lack of any IAS directions I decided to create my own. I am using a lab environment that consists of the following items.

Windows Server 2003 SP1 Active Directory
Windows Server 2003 SP1 IAS (Internet Authentication Services)
APC AP9617 Web/SNMP Management card with AOS Firmware v2.6.4
APC SmartUPS with SmartSlot

Before we can begin, IAS must be installed and registered in Active Directory.

 

We will need to be sure that the two devices are reachable. If both the APC units and your RADIUS server are on the same network segment then this is probably not an issue, but in an enterprise deployment it is very possible that these would be on different subnets. So let’s start by pinging our APC unit from the IAS server

C:>ping 10.0.2.38

Pinging 10.0.2.38 with 32 bytes of data:

Reply from 10.0.2.38: bytes=32 time=1ms TTL=128
Reply from 10.0.2.38: bytes=32 time=1ms TTL=128
Reply from 10.0.2.38: bytes=32 time=1ms TTL=128
Reply from 10.0.2.38: bytes=32 time=1ms TTL=128

Create a RADIUS Client

Once we have confirmed that the device is reachable we can create a RADIUS Client. In IAS right click on "RADIUS Clients" and select "New RADIUS Client"

The new RADIUS client wizard walks you though the process of adding your RADIUS device.

In the "Friendly name" field name the object "APC UPS" or anything that begins with "APC". This will be important when we create the Remote Access Policy later on.

Next, populate the "Client address (IP or DNS)" field with the IP address or DNS name of the APC Management card in the UPS and then click "Next".

 

On the following screen select "RADIUS Standard" from the Client-Vendor list. (APC is not in the list) Then enter a password the device will use to authenticate to IAS in the "Shared secret" field. In this case I am using the shared secret "APC". Confirm your shared secret and click "Finish". Do NOT check the box "Request must contain the Message Authenticator attribute".

 Note: Shared secrets are case sensitive.

 

Create a RADIUS Policy

Now we will need to create a Remote Access Policy. We will first create a policy that will allow Domain Adminis administrator access to the Web/SNMP card. To begin right click on "Remote Access Policy" and select "New Remote Access Policy"

When the New Remote Access Policy Wizard begins click "Next".

Select "Set up a custom policy" under "How do you want to set up this policy?" Then create a name for your policy. I have decided to name mine "APC Policy". Once you named your policy click "Next".

 

On the Policy Conditions page click "Add" and select "Windows-Groups" and click "Add" again.

In the Groups page click "Add" and select the Active Directory user group you want to have administrator access to your APC devices. In my case I have added the Domain Admins group. Click "OK" when you have finished adding groups.

 

 

Once again on the Policy Conditions page click the "Add" button and choose "Client-Friendly-Name" from the list and click "Add".

 

In the Client-Friendly-Name window type "APC*". This will ensure that this Remote Access Policy only applies to RADIUS clients whose name begins with "APC". This is something we defined above when we originally created the RADIUS client in IAS. When you are done click "OK"

 

Your Policy conditions should look similar to this.

 

Once you have double checked your settings click "Next". On the Permissions page select "Grant remote access permission" and click "Next"

 

On the Profile page click "Edit Profile" and go to the "Authentication" tab. Check the box labeled "Unencrypted authentication (PAP, SPAP). Uncheck all other boxes.

 

Next, click on the "Encryption" tab and clear all the checkboxes except "No encryption"

 

Lastly; click on the "Advanced" tab and remove all attributes that are listed, and then click "Add". Select "Service-Type" from the list and click "Add".

 

Under the "Attribute value" pull down select "Administrative" and click "OK".

 

Back at the Add Attribute page click "close" and you will be return to the profile window. It should look like this.

 

Click "Apply" then click "Ok".

 

At this point you should be back at the New Remote Access Policy Wizard. Click "Next", and then click "Finish".

 

You should now have a fully working RADIUS policy that will allow your Domain Admins administrator access to your APC Web/SNMP enabled UPS. You can try accessing the web interface or SSH/Telnet interface if you have that enabled. You can also create a "Device" policy if you like. Device access on an APC unit allows full control over the device the Web/SNMP card is attached to. For example, if you have the Web/SNMP card installed in a UPS, the device permission would allow the user to manage the UPS but it would not allow that user to modify settings related to the Web/SNMP card such as TCP/IP or SNMP settings. This could be helpful if you have a helpdesk staff that needed to check the estimated run time of the UPS, but you did not want them to modify any critical settings of the unit.

To create a RADIUS policy to allow the helpdesk staff access to your APC equipment with the Device permissions start by creating a new security group in Active Directory. I have named mine "Helpdesk". Add the users you would like to have Device permissions to your APC devices to your newly created security group.

 Once you have created your security group in Active Directory and added your users to the group you then need to create a new RADIUS Policy. Follow the same steps under "Create a RADIUS Policy" with the exceptions listed below. You do not need to create the RADIUS client again.

You will need to create your new RADIUS policy with a unique name. I have decided to name my Device access RADIUS Policy "APC Helpdesk"

 

When adding conditional policy attributes select "Windows-Groups" as you did before, but this time add your newly created group instead of Domain Admins.

 

Follow the instructions as they are written above until you add the "Service-Type" attribute. This attribute determines what access level the user will have once they have authenticated. "Administrative" would give the user Administrator access to the device. To give the user Device level access only, choose "Login" from the "Attribute value" pull down.

 

Continue to follow the directions above until you have finished creating your RADIUS Policy. When you have completed your Helpdesk staff should have Device level access to your APC RADIUS clients. 

So that’s how to manage access to your APC equipment using IAS in a nutshell.

 

if you’re blessed with the talent, utilize it to the fullest be true to yourself and stay humble. 

-Blackalicious

MSN: zmq503o1@hotmail.comAIM: zmq503o1Yahoo: zmq503o1ICQ: 3802219gTalk: jmorrill@gmail.comSkype: zmq503o1

 

Posted in Uncategorized | 13 Comments

Telnet & Authenticated SMTP

I recently had the misfortune of troubleshooting a problem with SMTP on a Microsoft Exchange server. This ordinarily would not have been so difficult but this SMTP server required authentication. I was disappointed with the lack of useful information provided to me by the traditional mail client  why SMTP was failing, so I decided to telnet to the SMTP server and see what was up. It was a little difficult at first to figure out the proper way of authenticating within a telnet session but after some research I was able to figure it out. Below is the process that I followed.
 
First telnet to the SMTP server on port 25
telnet smtp.domain.com 25
You should receive a message similar to the following
220 smtp.domain.com Microsoft ESMTP MAIL Service, Version: 6.0.3790.2499 ready at Mon, 2 Jan 2006 14:45:31 -0500
Next we will issue the command “EHELO” to begin our session.
EHLO
You should receive a similar response
250-smtp.domain.com Hello [10.1.2.32]
more… verbs
250 OK
NOTES:
ESMTP uses ehlo as a greeting – to establish an initial connection.
SMTP uses helo instead of ehlo.
250 – Means that the action which you requested, completed OK.  (Remember codes beginning with 2yz indicate success.
For simplicity, I have omitted the list of extra ESMTP verbs.
 
Now for the tricky part. Because this SMTP server requires authentication we will use the “AUTH LOGIN” command.
AUTH LOGIN
334 VXNlcm5hbWU6
The response of “VXNlcm5hbWU6” might seem a little strange, but after decoding from Base64 it all becomes clear.  “VXNlcm5hbWU6” is “Username:” Since we are being challenged in Base64, we will need to respond in Base64. You will need a Base64 encoder/decoder. I recommend http://www.dillfrog.com/tools/base-64_encode/
 
Enter your Base64 encoded username into your telnet session. I recommend copy/paste from the site above.
dXNlcm5hbWU=
334 UGFzc3dvcmQ6
From our Base64 decoder you will see that “UGFzc3dvcmQ6” means “Password:” Enter your password in Base64.

cGFzc3dvcmQ=
235 2.7.0 Authentication successful.

If you entered the correct username & password for a working account you should have successfully authenticated. Now we will compose the message. We begin by specifying who the message is from using “MAIL FROM”, then identifying who the message is to using “RCPT TO”. Here is an example.

 
MAIL FROM: username@domain.com
250 2.1.0 username@domain.com….Sender OK
RCPT TO: recipient@domain.com
250 2.1.5 recipient@domain.com
 
Now we can begin composing our message. We start with the “DATA” command.
DATA
354 Start mail input; end with <CRLF>.<CRLF>
If you wish to have a subject line enter “Subject:” then your subject line text. In order to conform to RFC 882 you must have two carriage returns (enter twice) after your subject line before composing the body of your message.
Subject: This is my subject
 
This is the body of my message
.
250 2.6.0 <
SMTPvgs0LtQ5A0JB0i00002712@SMTP.domain.com> Queued mail for delivery
After your subject line and two carriage returns you can begin typing your message. When you are done composing, type a period “.” on a line by itself and press enter. This will denote the end of your message and deliver the email. Once your message is sent you can issue the “QUIT” command to end your SMTP session with the server.
QUIT
221 2.0.0 smtp.domain.com Service closing transmission channel
Here is the whole sequence without interruption
telnet smtp.domain.com 25
220 smtp.domain.com Microsoft ESMTP MAIL Service, Version: 6.0.3790.2499 ready at Mon, 2 Jan 2006 14:45:31 -0500
EHLO
250-smtp.domain.com Hello [10.1.2.32]
more… verbs
250 OK
AUTH LOGIN
334 VXNlcm5hbWU6
dXNlcm5hbWU=
334 UGFzc3dvcmQ6
cGFzc3dvcmQ=
235 2.7.0 Authentication successful.
MAIL FROM:
username@domain.com
250 2.1.0 username@domain.com….Sender OK
RCPT TO:
recipient@domain.com
250 2.1.5 recipient@domain.com
DATA 
354 Start mail input; end with <CRLF>.<CRLF>
Subject: This is my subject
This is the body of my message
.
250 2.6.0 <
SMTPvgs0LtQ5A0JB0i00002712@SMTP.domain.com> Queued mail for delivery
QUIT
221 2.0.0 smtp.domain.com Service closing transmission channel
Enjoy
 

MSN: zmq503o1@hotmail.comAIM: zmq503o1Yahoo: zmq503o1ICQ: 3802219gTalk: jmorrill@gmail.comSkype: zmq503o1

 
Posted in Computers and Internet | 35 Comments

Fun with Private VLANs

Private VLANs have some level of mystique which is probably due to its confusing and complicated implementation. The easiest thing to understand about a Private-VLAN is the end result.

What is the purpose of private VLANs?

Private VLANs are more or less what the name implies. The illusion is that each physical access port on the network is on its very own individual VLAN. Of course this is not really the case; as that would be extremely difficult to build, and almost impossible to maintain.  Instead, Private VLANs provide all the security of separate individual VLANs for your access ports, with the ease and simplicity of managing a traditional LAN.

Ports within a Private VLAN are broken up into three categories, isolated, community, and promiscuous. Hosts connected to a promiscuous port can talk to any port, including isolated and community ports. Promiscuous ports are usually reserved for router interfaces, printers and servers on the same VLAN which all clients need access to. Hosts connected to isolated ports can not communicate with anyone else on the same VLAN with the exception of promiscuous ports. Isolated ports are most commonly used in Schools, dormitories, hotels, libraries and other public locations. Hosts connected to community ports may communicate to other common community ports or promiscuous ports but may not talk to isolated ports. More on community ports later.

I am going to give an example of how to create and configure a private VLAN on a network that consists of two Cisco Catalyst 3750’s both connected to a common 6500 running CatOS with two MSFCs.

The scenario is this. A local convention center is going to be providing internet access to each booth. The network administrator wishes to keep the convention traffic separate from the administrative systems, and also ensure that the users who will be renting the booths will be protected from the users in surrounding booths.

I will start by creating a new VLAN on the 6500. This VLAN will be the Primary VLAN. Every private VLAN begins with a primary VLAN. This primary VLAN is basicly your regular VLAN, in which the isolated and community VLANs will be associated. The Primary VLAN is also the only VLAN you need to be concerned about when configuring your MSFC.

6500# (enable) set vlan 11 name Convention_Primary_VLAN pvlan-type primary

Next I will create an isolated VLAN, I know this sounds confusing but trust me, this is how it works.

6500# (enable)set vlan 111 name Convention_Isolated_VLAN pvlan-type isolated

If you plan on using a community private VLAN create it now with “pvlan-type” of “community”

Now we need to associate the isolated VLAN and any community VLANs that are created with the primary VLAN.

6500# (enable) set pvlan 11 111

Lastly we need to create a VLAN mapping for the MSFCs to allow the traffic from the isolated VLAN and any community VLANs that were created to route beyond the local subnet.

6500# (enable) set pvlan mapping 11 111 15/1
6500# (enable) set pvlan mapping 11 111 16/1

If you are running in Native IOS mode you must go to each VLAN and create a private VLAN mapping

6500# config t
6500(config)#interface vlan 11
6500(config-if)#private-vlan mapping 111

If you have not already done so, add the newly created VLANs to the trunk port interfaces that serve as the uplink to the edge switches.

6500# (enable) set trunk 3/9 11,111
6500# (enable) set trunk 3/10 11,111

If you have any access ports on the 6500 you wish to add to the private VLAN you can do so now.

6500# (enable) set pvlan 11 111 9/22

That’s it for the CatOS side. On the MSFC you will need to create a VLAN interface for the primary VLAN. No interfaces should be created for your isolated or community VLANs.

MSFC# config terminal
MSFC(config)# interface vlan 11
MSFC(config-if)# description Convention_VLAN
MSFC(config-if)# ip address 10.0.1.1 255.255.255.0

Nothing additional needs to be configured on the MSFC.

We can now turn our attention to the edge switches. In this case I am using a Cisco Catalyst 3750. We will begin by creating the primary and isolated VLANs on the switch. (note VTP mode must be “transparent” for private VLANs to function properly)

3750# config terminal
3750#(config)# vtp mode transparent
3750#(config)# vlan 11
3750#(config-vlan)# name Convention_Primary_VLAN
3750#(config-vlan)# private-vlan primary
3750#(config-vlan)# vlan 111
3750#(config-vlan)# name Convention_Isolated_VLAN
3750#(config-vlan)# private-vlan isolated

Now we must associate the isolated VLAN with the primary VLAN.

3750#(config-vlan)# vlan 11
3750#(config-vlan)# private-vlan association 111

Lastly we need to add interfaces on the switch to the private VLAN. In this case we will be adding ports 13 – 24.

3750#(config-vlan)# interface range fa1/0/13 -24
3750#(config-if-range)# switchport private-vlan host-association 11 111
3750#(config-if-range)# switchport mode private-vlan host

Duplicate this process on the second 3750 and begin testing. Begin by plugging a host into any of the isolated ports configured on the 3750. Assign an IP address or get one from a DHCP server. I will configure my first machine with the IP address of 10.0.1.5/24. Next attach a host to an isolated port on the second 3750 or the optional isolated port we configured on the 6500. Give the host an IP address, or obtain one from a DHCP server. Mine will be 10.0.1.6/24.

You can now begin testing the configuration. Start by trying to ping each host from the other. If you look at the ARP table on the each of the hosts you will notice that you are unable to ARP the other host. You can try IPX and Appletalk if you wish. The results will be the same. The only interface in the configuration above that the hosts should be able to ping is the router.

Community VLANs

Community VLANs are created the exact same way as isolated VLANs. What makes them powerful is that you can have multiple community VLANs associated with a single primary VLAN. An example of its use would be if you had a corporate P2P application and you wanted all the machines in marketing to be able to share files with each other but not with sales. You also wanted the people in sales to share files with each other but not with marketing. Community VLANs allow you to do this within the same IP subnet.

Notes
 
It should be said that if you are running Native IOS on your 6500 and you configure private VLANs for the first time Sticky ARP is enabled by default. Sticky ARP is a Cisco security feature designed to be used with static IP addressing to prevent ARP spoofing. This means that once an ARP entry is created on the router this entry becomes permanent until the 6500 is rebooted. This is far from ideal in a DHCP environment. You will find clients making DHCP requests, receiving DHCP offers and accepting these offers but unable to access the network because the IP address the client was assigned once belonged to a different MAC address. With that said, Sticky ARP should be disabled either globally or on the specific VLANs that you will be running Private VLANs on.

To disable Sticky Arp globally:

6500# config terminal
6500(config)#no ip sticky-arp

To disable Stick Arp on a specific VLAN

6500# config terminal
6500(config)# interface Vlan11
6500(config-if)# ip sticky-arp ignore

If you have security concerns about disabling Sticky ARP I strongly suggest you look into IP source guard which is part of Cisco’s DHCP Snooping protection. More on that later.

 Yahoo: zmq503o1MSN: zmq503o1@hotmail.comAIM: zmq503o1ICQ: 3802219Google Talk: jmorrill@gmail.comSkype: zmq503o1

 

Update: I came across the following error the other day and thought it was note worthy since it was related to the private VLAN configuration.

%PM-SP-3-ERR_INCOMP_PORT: 9/21 is set to inactive because 9/1 is a dynamic trunk port

It seems that if you have an interface configured for a private vlan "9/21", and another interface configured as a trunk "9/1" sharing the same ASIC you will run into a similar error message. Although this is not particularly note worthy I should point out that in my configuration 9/1 is not explicitly defined as a trunking interface. In fact it was being used as an access port, not a trunk port.

So what’s the problem then?

Well all interfaces by default are configured as dynamic trunk ports. This can be seen by running the IOS command

IOS-6509# show interface trunk module 9

This will display the trunking status of the interfaces on module 9.

Port          Mode         Encapsulation  Status        Native vlan
Gi9/1         desirable    negotiate      other         1

For private-vlan interfaces to work properly with non-private-vlan interfaces on the same switch, within the same ASIC, dynamic trunking must be disabled on all non-private-vlan interfaces.

IOS-6509# config terminal
IOS-6509(config)# interface g9/1
IOS-6509(config-if)# switchport mode access

For instance, on a WS-X6548-GE-TX ports 1 though 24 share an ASIC. If you were to have interface 1 configured as a typical access port, and interface 2 configured for a private-vlan, you must disable dynamic trunking on port 1.

Check the configuration and make sure trunking is disabled.

IOS-6509# show interface trunk module 9

Port          Mode         Encapsulation  Status        Native vlan
Gi9/1         off          802.1q         not-trunking  1

 

Posted in Computers and Internet | 29 Comments

Quest for the Ultimate free TFTP Server

I remember a time when every network equipment manufacturer offered a free Windows TFTP Server for download. There also were tons of freeware TFTP Servers available all over the net created by individuals. That time seems to have passed. It’s getting more and more difficult to find a decent TFTP Server these days. Even Cisco, the largest supplier of network equipment in the world no longer offers a free TFTP Server.

My requirements are probably high. The TFTP Server must run as a Windows service; must support very large file transfers, and most importantly, must be free.

For years I had been using 3Com’s TFTP Service. This met my needs at the time. It runs as a Windows Service, and of course, it’s free. To be completely honest this is an excellent free TFTP Server, but the maximum file size supported is 32Meg. Any file transferred that is larger then 32Meg dies at the 32meg mark and does not continue. Needless to say it was time to bid farewell to my Trusty 3Com TFTP Server.

3Com also offers the 3CServer which, other then being extremely old is truly a fantastic TFTP Server. In addition to being a TFTP Server, 3CDaemon is also a Syslog Server, FTP Server and TFTP Client. This is an excellent Swiss army knife of network tools, but alas the 3CDaemon does not run as a Windows service, nor does it support files larger then 32Meg.

If you Google "Free TFTP Server" your first hit is the SolarWinds TFTP Server. I happen to own the SolarWinds Engineers Edition which also contains this utility. To be honest the same version of the SolarWinds TFTP Server that you download for free is what comes with the Engineers Edition.

This was by far the worst of the free TFTP Servers I tried. Not only does it not run as a Windows Service (which could be resolved following the directions from one of my previous blog entries) but the file size limitation is a ridiculous 16meg. You would be better off running either of the free 3Com TFTP Servers over this crappy one from SolarWinds.

Finally I happened upon TFTP Nirvana. Most people are not aware that Windows 2000 Server and Windows Server 2003 both come with a great TFTP server. The only trouble is you need to install RIS (Remote Installation Services) to get it. The Windows TFTP Server seems to meet all of my criteria. It’s free, as in "included in the OS". It runs as a Windows service, and I have been unable to find any file size limit. The only Trouble is that you need to install RIS to get the TFTP Service. Morgan Simonsen has eliminated the need to install RIS with his custom TFTP installation script. TFTPDaemon v1.1 contains everything you need to install Microsoft’s TFTP service on any Windows 2k, XP, 2k3 machine.

 

Yahoo: zmq503o1MSN: zmq503o1@hotmail.comAIM: zmq503o1ICQ: 3802219Google Talk: jmorrill@gmail.comSkype: zmq503o1

Posted in Computers and Internet | 45 Comments