A very newbie to linux. have to do this final year project on beowulf cluster with linux 6.1os. using a myrinet 8 port switch to connect master to slave nodes. Question1:
how do i set up internet connection via network point in my universtiy? Question 2: i think i might have altered the prompt that orignally read~master to ~local host how do i change it back? Question 3: How do i run or send jobs to the other nodes? what do ineed? Qustion 4. How is benchmark rated?
thank you all for ur kind help!!!
Beowulf_Ghost
12-07-2000, 04:48 AM
First off; JESEUS CRISPIES!
now that that's out of the way...
answers;
1)You need to put another NIC in you master node and get an IP from your school. I'm assuming you want to log into the master node remotely, and issue commands to the cluster. Basically, you need to set up one NIC for talking to the cluster, and the other NIC for talking to the outside world. Configuring this depends on what distor you are using.
2)The name the prompt gives you is the name of the machine you are using. Usually it changes when you edit hostname in you networking preferences.
3)MPI, PVM, Mosix? You can always send commands to the other nodes by using rsh (provided your security is lax enough).
4)The best thing you should do, is go to www.beowulf-underground.org (http://www.beowulf-underground.org) and check out the software they have. They have a bunch of benchmarking tool, admin tools, and apps compiled to run on clusters.
Good Luck
Fandelem
12-07-2000, 07:00 AM
I'm hoping that this project isn't due for a very, very long time or you have a bunch of other groupies who know about this stuff.. :}
Otherwise you're going to be spending a few sleepless nights living off caffeine learning a great deal of information in a short time frame http://www.linuxnewbie.org/ubb/biggrin.gif
guy above me has the right idea. listen to him :}
klamath
12-07-2000, 03:29 PM
How exactly does a total Linux newbie have access to a Linux beowulf cluster? As Fandelem said, get ready for some fun!
------------------
- Klamath
Get my GnuPG Key Here (http://klamath.dyndns.org/mykey.asc)
Looking for an open source project to contribute to? Check out the Better Bulletin Board (http://bbb.sourceforge.net)
Beowulf_Ghost
12-07-2000, 04:09 PM
Beowulf Clusters are easier to put together then most people think. All most any Linux geek can put together an academic level cluster for real cheap. Get you hands on some old P100,s some NIC's and a hub. If you have more then 1 computer in your house, and they are networked together, you're half way there.
Were clusters get real hard, is when you have to make a production system. A cluster that will actually preform some task with reasonable performance. These are the guys with 200 or so Athlon/PIII/Alpha boxes on gigabit ethernet.
But back to cheap clusters;
An execelent exanple of some people that built a cluster out of what ever was at hand can be found at; http://www.tri-gon.net/bolo.htm
Back to helping Linuxduh;
I'm assuming "linux 6.1os" is Red Hat 6.1. If so, you're in luck. Red Hat is one of the more popular distros for building clusters with. And because of that, their is plenty of documentation on it.
Another good site for Beowulf info; http://www.beowulf.org
They have a list of other clusters, and a mailing list to post questions to. They also have links to other Beowulf related sites, and places to get software. Both free and commercial.
I am planning it out for my house. I am VERY interested in doing it, but it'l be slower going for me, as it is just a side-projuect kinda thing.
linuxduh
12-07-2000, 10:26 PM
HI
i got the hostname figured out ;p. and u are right as in i want to do remote login via masternode. I really have no idea how to go about doing the NIC . I was told that the ip can be dynamically assigned( but i dont knwo how to do this) So very thank ful for ur help , can u pls further give me an idea as to how to setup cos i am really at a loss here.
Thank u so much
sincerely
ld
Originally posted by Beowulf_Ghost:
First off; JESEUS CRISPIES!
now that that's out of the way...
answers;
1)You need to put another NIC in you master node and get an IP from your school. I'm assuming you want to log into the master node remotely, and issue commands to the cluster. Basically, you need to set up one NIC for talking to the cluster, and the other NIC for talking to the outside world. Configuring this depends on what distor you are using.
2)The name the prompt gives you is the name of the machine you are using. Usually it changes when you edit hostname in you networking preferences.
3)MPI, PVM, Mosix? You can always send commands to the other nodes by using rsh (provided your security is lax enough).
4)The best thing you should do, is go to www.beowulf-underground.org (http://www.beowulf-underground.org) and check out the software they have. They have a bunch of benchmarking tool, admin tools, and apps compiled to run on clusters.
Good Luck
linuxduh
12-07-2000, 10:28 PM
HI Fandelum
sigh... its due in early jan /feb. and no there is no one who knows this stuff .. sigh.. can u like also give me some clear instructions... i really am sorry to say this but even the tech support staff gave up....
sincerely
linuxduh
Originally posted by Fandelem:
I'm hoping that this project isn't due for a very, very long time or you have a bunch of other groupies who know about this stuff.. :}
Otherwise you're going to be spending a few sleepless nights living off caffeine learning a great deal of information in a short time frame http://www.linuxnewbie.org/ubb/biggrin.gif
guy above me has the right idea. listen to him :}
linuxduh
12-07-2000, 10:28 PM
Hi
u dont know the kind of 'luck'i have....
btw can u help?
sincerely
ld
Originally posted by klamath:
How exactly does a total Linux newbie have access to a Linux beowulf cluster? As Fandelem said, get ready for some fun!
linuxduh
12-07-2000, 10:30 PM
Hi
u are right ! but i still donno how to get up the network for internet access cos the documentation they have is for dialup and all that fiddling has resulted in httpd failed at startup. also my eth0 and eth1 cannot be brought up . can u advise?
thanku
sincerely
ld
Originally posted by Beowulf_Ghost:
Beowulf Clusters are easier to put together then most people think. All most any Linux geek can put together an academic level cluster for real cheap. Get you hands on some old P100,s some NIC's and a hub. If you have more then 1 computer in your house, and they are networked together, you're half way there.
Were clusters get real hard, is when you have to make a production system. A cluster that will actually preform some task with reasonable performance. These are the guys with 200 or so Athlon/PIII/Alpha boxes on gigabit ethernet.
But back to cheap clusters;
An execelent exanple of some people that built a cluster out of what ever was at hand can be found at; http://www.tri-gon.net/bolo.htm
Back to helping Linuxduh;
I'm assuming "linux 6.1os" is Red Hat 6.1. If so, you're in luck. Red Hat is one of the more popular distros for building clusters with. And because of that, their is plenty of documentation on it.
Another good site for Beowulf info; http://www.beowulf.org
They have a list of other clusters, and a mailing list to post questions to. They also have links to other Beowulf related sites, and places to get software. Both free and commercial.
milanuk
12-07-2000, 11:04 PM
Originally posted by linuxduh:
Hi
u are right ! but i still donno how to get up the network for internet access cos the documentation they have is for dialup and all that fiddling has resulted in httpd failed at startup. also my eth0 and eth1 cannot be brought up . can u advise?
thanku
sincerely
ld
This is bizarre. Here I am on my little home LAN w/ at best a 21.6k connection to the Net, trying to explain networking under linux to someone w/ access to a university cluster. What's wrong w/ this picture? http://www.linuxnewbie.org/ubb/wink.gif
When they told you the NIC stuff could be set up dynamically, did the term 'DHCP' or 'Dynamic Host Configuration Protocol' come up at all? This might make things a bit easier, as it is a method for using one machine to tell all the other machines where to look for the internet gateway, nameservers, etc. I'm not exactly sure what you mean by the 'network point' at the university? The gateway/firewall/router btwn this little network and the rest of the college and/or internet? If so, you need that address so you can tell the machine where to send data not destined for the local network. I'm not sure, since I don't have OpenSSH installed yet on my Debian box here, so I can't reach the boy's RedHat 6.2 box (and I'm too lazy to walk upstairs and login manually http://www.linuxnewbie.org/ubb/wink.gif ), but I believe if you get to an xterm or console window and 'su' to root, you could type 'setup' and that should present you w/ a menued system to the tools used during a non-GUI installation to configure things like the mouse, X, and hopefully (can't remember for sure) the network.
Seriously, though, you would be well advised to run, not walk, to the Linux Documentation Project. There are many guides, HOWTO's, and other documents to help you pick things up. the 'Rute' guide is highly recommended, but you'll probably just want to skim it at this point, due to lack of time. Another one you might want is the Unix-and-Internet-Fundamentals-HOWTO. This explains the Internet and networking from a *nix perspective, which can be decidedly different from the M$ view, if that's what you're used to.
Monte
Beowulf_Ghost
12-08-2000, 03:18 AM
In order for me, or any one else on this board, to help you, we need to know exactly what you are running.
What kind of computers, what kind a networking equipment, and esespecialy what Linux distrobution you are running (Red Hat, Debian, etc..) and any patches you might have applied to the kernel (like Mosix).
I'll continue under the assumption that you are running Red Hat 6.1. If I remember right, ifconfig is the utility used for configuring network interfaces in Red Hat (any Red Hat user is free to correct me on this).
"ip can be dynamically assigned";
This is called DHCP. When you configure your networking, you usualy have a choice of choosing you IP address, or you can have one given to you. DHCP is the method by with an IP is dynamically assigned to you. Some where st your school, there is a DHCP server that gives IP address, and other info, to any machine asks. So when you configure the NIC that will talk to the school, don't assign the address your self. Tell it to use DHCP.
You mentioned that you messed up something, and httpd won't come up. Unless you really need to use the master node as a web server, you should just turn off httpd. The purpose of a Beowulf cluster is performance above all else. High performance, low availability.
That's all I have for now. I'll see if I can get a Red Hat box running, and post some better instructions tommorow.
Isn't the clustering technology a kernel-specific function (not distro-specific)?
linuxduh
12-09-2000, 10:10 PM
hi the specs are as follows
have total of 5 nodes running on rh6.1 with linda. have a myrinet 8 ports switch connected to the nodes.
Problem:1) Unable to ping other nodes from master node. I can send to the current working nodes but cant sense the other nodes
2) Would like to know how to start sending jobs to the other nodes(once the above problekm is solved) . Would also like to know how exactly do i setup dhcp.
Pls do help me on this and also include specific instructions like which files to alter etc.. I am really very grateful to this forum and the ppl who come here.
Thank you all
sincerely
linuxduh
Beowulf_Ghost
12-10-2000, 04:09 PM
From what you've told me, this is what I have so far.
http://65.0.85.84/topology.jpg
I'll help you were I can, but there are two things that I can't help you with;
1) Myrinet: I've never used myrinet. It's a little out of my price range. I understand networking in general, but I can't help you with _specifics_ of myrinet. Possibly some else on this board can.
2) Linda: I've read a little about Linda, but I've never used it.
"Problem:1) Unable to ping other nodes from master node. I can send to the current working nodes but cant sense the other nodes."
I'm not sure what you mean by this.
For Red Hat 6.1, the utility used for configuring network interfaces is, netconfig. When you start netconfig, the first thing it you do is either, enter in your IP and DNS by hand, or it lets you choose DHCP. You will want to choose DHCP for your conection to the Universty.
As for the other nodes. Make sure that rheir networking is up. Try pinging the master node from one of the slave nodes. If the master node responds, then that means the slave node has a problem responding to pings. Basically go though each node, and ping the other nodes.
Go through the /etc/inetd.conf file on all your nodes, and make sure that all the services you want are on. To make sure you can use rsh, take out an # makers in front of the line begining with shell.
linuxduh
12-10-2000, 09:35 PM
HI!
Yes!!! u are right in the setup!!Thank gosh!!
ok. i tried to ping each node to find out which other nodes they can ping.
problem 1:
master node: host unreachable for all (master &node2-5)
node2: only node2 can be ping rest is unreachable host
same for node3-5
in conclusion i can ping the respective nodes when i log in to them individually but not wheni want to ping the other nodes from it. the masternode is even worse i can't even ping it while at it.
problem 2:hmm i dont quite get what u mean by the /etc/inetd.conf i dont really know what services i need how would i know if the services are on?
problem 3:
Also at startup ihave the problem of http failed. what could be wrong?
THank you so m uch for ur generous help!
sincerely
linuxduh
Beowulf_Ghost
12-11-2000, 06:51 AM
Here is a copy of my /etc/inetd.conf file;
Begin here---
#
# inetd.conf This file describes the services that will be available
# through the INETD TCP/IP super server. To re-configure
# the running INETD process, edit this file, then send the
# INETD process a SIGHUP signal.
#
# Version: @(#)/etc/inetd.conf 3.10 05/27/93
#
# Authors: Original taken from BSD UNIX 4.3/TAHOE.
# Fred N. van Kempen, <waltje@uwalt.nl.mugnet.org>
#
# Modified for Debian Linux by Ian A. Murdock <imurdock@shell.portal.com>
#
# Modified for RHS Linux by Marc Ewing <marc@redhat.com>
#
# <service_name> <sock_type> <proto> <flags> <user> <server_path> <args>
#
# Echo, discard, daytime, and chargen are used primarily for testing.
#
# To re-read this file after changes, just do a 'killall -HUP inetd'
#
#echo stream tcp nowait root internal
#echo dgram udp wait root internal
#discard stream tcp nowait root internal
#discard dgram udp wait root internal
#daytime stream tcp nowait root internal
#daytime dgram udp wait root internal
#chargen stream tcp nowait root internal
#chargen dgram udp wait root internal
#time stream tcp nowait root internal
#time dgram udp wait root internal
#
# These are standard services.
#
#ftp stream tcp nowait root /usr/sbin/tcpd in.ftpd -l -a
#telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd
#
# Shell, login, exec, comsat and talk are BSD protocols.
#
#shell stream tcp nowait root /usr/sbin/tcpd in.rshd
#login stream tcp nowait root /usr/sbin/tcpd in.rlogind
#exec stream tcp nowait root /usr/sbin/tcpd in.rexecd
#comsat dgram udp wait root /usr/sbin/tcpd in.comsat
#talk dgram udp wait nobody.tty /usr/sbin/tcpd in.talkd
#ntalk dgram udp wait nobody.tty /usr/sbin/tcpd in.ntalkd
#dtalk stream tcp wait nobody.tty /usr/sbin/tcpd in.dtalkd
#
# Pop and imap mail services et al
#
#pop-2 stream tcp nowait root /usr/sbin/tcpd ipop2d
#pop-3 stream tcp nowait root /usr/sbin/tcpd ipop3d
#imap stream tcp nowait root /usr/sbin/tcpd imapd
#
# The Internet UUCP service.
#
#uucp stream tcp nowait uucp /usr/sbin/tcpd /usr/lib/uucp/uucico -l
#
# Tftp service is provided primarily for booting. Most sites
# run this only on machines acting as "boot servers." Do not uncomment
# this unless you *need* it.
#
#tftp dgram udp wait root /usr/sbin/tcpd in.tftpd
#bootps dgram udp wait root /usr/sbin/tcpd bootpd
#
# Finger, systat and netstat give out user information which may be
# valuable to potential "system crackers." Many sites choose to disable
# some or all of these services to improve security.
#
finger stream tcp nowait nobody /usr/sbin/tcpd in.fingerd
#cfinger stream tcp nowait root /usr/sbin/tcpd in.cfingerd
#systat stream tcp nowait guest /usr/sbin/tcpd /bin/ps -auwwx
#netstat stream tcp nowait guest /usr/sbin/tcpd /bin/netstat -f inet
#
# Authentication
#
auth stream tcp wait root /usr/sbin/in.identd in.identd -e -o
#
# End of inetd.conf
Notice how some of the lines have a # infront of them. This is commenting out a line. When Linux's networking services turn on during boot, they read this file. This file tells Linux what services to run, and how to run them. When Linux sees a line starting with a #, it ignores that line, and doesn't start that service. The only line that aren't commented out in my /etc/inetd.conf are "auth" and "finger". That means that the only services my computer runs are auth and finger. IT won't run FTP, or telnet.
On your nodes, you will wan't to make sure their are no # marks infront of the shell line. This alows you to run rsh (remote shell). rsh allows you to send commands to other computers.
http problem;
Unless you really need to use the master node as a web server, I would worry about http later. If other computers can't even ping the master, then they aren't gonna get web page from it either. One problem at a time.
back to pining;
I'm not sure I understand which nodes are working. So do this;
(from hear on, we'll just call the master node "node 1")
list each node, and what nodes it gets pings back from
example;
node1;
node2;345
node3;2
node4;2
node5;2
The example above says that node 2 will ping node 3, node 4 and node 5. And node 3 eill only ping node 3.
This will help me figure out where your point of failure is.
Something else you should do. Try telnet'ing to the other nodes. It may just be a problem with ping.
linuxduh
12-12-2000, 11:16 PM
Hi
i can't ping any of the node2-5 from node1 others i can ping amongs them(ie.node2-5 exculsive of node1) i cannot telnet also from the same nodes as above. just hangs there . i also notice that my eth0 is down funny as the connection to the network card is green lights.
what now???
thanks ;]
linuxduh
12-12-2000, 11:18 PM
oh yah
now i have added an additional 3com hub to the existing cluster so besides via myrinet switch all the nodes are also connected via hub. still can';t ping.. sad...;~
sincerely
ld
justlinux.com
Copyright Internet.com Inc. All Rights Reserved.