The fileserver command is not normally issued at the command shell prompt, but rather placed into a database server machine's /etc/openafs/BosConfig file with the bos create command. If it is ever issued at the command shell prompt, the issuer must be logged onto a file server machine as the local superuser "root".
The File Server creates the /var/log/openafs/FileLog log file as it initializes, if the file does not already exist. It does not write a detailed trace by default, but the -d option may be used to increase the amount of detail. Use the bos getlog command to display the contents of the log file.
The command's arguments enable the administrator to control many aspects of the File Server's performance, as detailed in OPTIONS. By default the fileserver command sets values for many arguments that are suitable for a medium-sized file server machine. To set values suitable for a small or large file server machine, use the -S or -L flag respectively. The following list describes the parameters and corresponding argument for which the fileserver command sets default values, and the table below summarizes the setting for each of the three machine sizes.
The default values are:
Parameter (Argument) Small (-S) Medium Large (-L) --------------------------------------------------------------------- Number of LWPs (-p) 6 9 128 Number of cached dir blocks (-b) 70 90 120 Number of cached large vnodes (-l) 200 400 600 Number of cached small vnodes (-s) 200 400 600 Maximum volume cache size (-vc) 200 400 600 Number of callbacks (-cb) 20,000 60,000 64,000 Number of Rx packets (-rxpck) 100 150 200
To override any of the values, provide the indicated argument (which can be combined with the -S or -L flag).
The amount of memory required for the File Server varies. The approximate default memory usage is 751 KB when the -S flag is used (small configuration), 1.1 MB when all defaults are used (medium configuration), and 1.4 MB when the -L flag is used (large configuration). If additional memory is available, increasing the value of the -cb and -vc arguments can improve File Server performance most directly.
By default, the File Server allows a volume to exceed its quota by 1 MB when an application is writing data to an existing file in a volume that is full. The File Server still does not allow users to create new files in a full volume. To change the default, use one of the following arguments:
By default, the File Server implicitly grants the "a" (administer) and "l" (lookup) permissions to system:administrators on the access control list (ACL) of every directory in the volumes stored on its file server machine. In other words, the group's members can exercise those two permissions even when an entry for the group does not appear on an ACL. To change the set of default permissions, use the -implicit argument.
The File Server maintains a host current protection subgroup (host CPS) for each client machine from which it has received a data access request. Like the CPS for a user, a host CPS lists all of the Protection Database groups to which the machine belongs, and the File Server compares the host CPS to a directory's ACL to determine in what manner users on the machine are authorized to access the directory's contents. When the pts adduser or pts removeuser command is used to change the groups to which a machine belongs, the File Server must recompute the machine's host CPS in order to notice the change. By default, the File Server contacts the Protection Server every two hours to recompute host CPSs, implying that it can take that long for changed group memberships to become effective. To change this frequency, use the -hr argument.
The File Server stores volumes in partitions. A partition is a filesystem or directory on the server machine that is named "/vicepX" or "/vicepXX" where XX is ``a'' through ``z'' or ``aa'' though ``iv''. Up to 255 partitions are allowed. The File Server expects that the /vicepXX directories are each on a dedicated filesystem. The File Server will only use a /vicepXX if it's a mountpoint for another filesystem, unless the file "/vicepXX/AlwaysAttach" exists. The data in the partition is a special format that can only be access using OpenAFS commands or an OpenAFS client.
The File Server generates the following message when a partition is nearly full:
No space left on device
This command does not use the syntax conventions of the AFS command suites. Provide the command name and all option names in full.
Do not specify both the -spare and -pctspare arguments. Doing so causes the File Server to exit, leaving an error message in the /var/log/openafs/FileLog file.
Options that are available only on some system types, such as the -m and -lock options, appear in the output generated by the -help option only on the relevant system type.
Currently, the maximum size of a volume is 2 terabytes (2^31 bytes) and the maximum size of a /vicepX partition on a fileserver is 2^64 kilobytes. The maximum partition size in releases 1.4.7 and earlier is 2 terabytes (2^31 bytes). The maximum partition size for 1.5.x releases 1.5.34 and earlier is 2 terabytes as well.
The maximum number of directory entries is 64,000 if all of the entries have names that are 15 octets or less in length. A name that is 15 octets long requires the use of only one block in the directory. Additional sequential blocks are required to store entries with names that are longer than 15 octets. Each additional block provides an additional length of 32 octets for the name of the entry. Note that if file names use an encoding like UTF-8, a single character may be encoded into multiple octets.
In real world use, the maximum number of objects in an AFS directory is usually between 16,000 and 25,000, depending on the average name length.
The maximum number of threads can differ in each release of OpenAFS. Consult the OpenAFS Release Notes for the current release.
File Server is running at I<time>.
The throttling behaviour can cause issues especially for some versions of the Windows OpenAFS client. When using Windows Explorer to navigate the AFS directory tree, directories with only ``look'' access for the current user may load more slowly because of the throttling. This is because the Windows OpenAFS client sends FetchStatus calls one at a time instead of in bulk like the Unix Open AFS client.
Setting the threshold to 0 disables the throttling behavior. This option is available in OpenAFS versions 1.4.1 and later.
% bos create -server fs2.abc.com -instance fs -type fs \ -cmd "/usr/lib/openafs/fileserver -pctspare 10 \ -L" /usr/lib/openafs/volserver /usr/lib/openafs/salvager
Process Signal OS Result --------------------------------------------------------------------- File Server XCPU Unix Prints a list of client IP Addresses. File Server USR2 Windows Prints a list of client IP Addresses. File Server POLL HPUX Prints a list of client IP Addresses. Any server TSTP Any Increases Debug level by a power of 5 -- 1,5,25,125, etc. This has the same effect as the -d XXX command-line option. Any Server HUP Any Resets Debug level to 0 File Server TERM Any Run minor instrumentation over the list of descriptors. Other Servers TERM Any Causes the process to quit. File Server QUIT Any Causes the File Server to Quit. Bos Server knows this.
The basic metric of whether an AFS file server is doing well is the number of connections waiting for a thread, which can be found by running the following command:
% rxdebug <server> | grep waiting_for | wc -l
Each line returned by "rxdebug" that contains the text ``waiting_for'' represents a connection that's waiting for a file server thread.
If the blocked connection count is ever above 0, the server is having problems replying to clients in a timely fashion. If it gets above 10, roughly, there will be noticable slowness by the user. The total number of connections is a mostly irrelevant number that goes essentially monotonically for as long as the server has been running and then goes back down to zero when it's restarted.
The most common cause of blocked connections rising on a server is some process somewhere performing an abnormal number of accesses to that server and its volumes. If multiple servers have a blocked connection count, the most likely explanation is that there is a volume replicated between those servers that is absorbing an abnormally high access rate.
To get an access count on all the volumes on a server, run:
% vos listvol <server> -long
and save the output in a file. The results will look like a bunch of vos examine output for each volume on the server. Look for lines like:
40065 accesses in the past day (i.e., vnode references)
and look for volumes with an abnormally high number of accesses. Anything over 10,000 is fairly high, but some volumes like root.cell and other volumes close to the root of the cell will have that many hits routinely. Anything over 100,000 is generally abnormally high. The count resets about once a day.
Another approach that can be used to narrow the possibilities for a replicated volume, when multiple servers are having trouble, is to find all replicated volumes for that server. Run:
% vos listvldb -server <server>
where <server> is one of the servers having problems to refresh the VLDB cache, and then run:
% vos listvldb -server <server> -part <partition>
to get a list of all volumes on that server and partition, including every other server with replicas.
Once the volume causing the problem has been identified, the best way to deal with the problem is to move that volume to another server with a low load or to stop any runaway programs that are accessing that volume unnecessarily. Often the volume will be enough information to tell what's going on.
If you still need additional information about who's hitting that server, sometimes you can guess at that information from the failed callbacks in the FileLog log in /var/log/afs on the server, or from the output of:
% /usr/afsws/etc/rxdebug <server> -rxstats
but the best way is to turn on debugging output from the file server. (Warning: This generates a lot of output into FileLog on the AFS server.) To do this, log on to the AFS server, find the PID of the fileserver process, and do:
kill -TSTP <pid>
where <pid> is the PID of the file server process. This will raise the debugging level so that you'll start seeing what people are actually doing on the server. You can do this up to three more times to get even more output if needed. To reset the debugging level back to normal, use (The following command will NOT terminate the file server):
kill -HUP <pid>
The debugging setting on the File Server should be reset back to normal when debugging is no longer needed. Otherwise, the AFS server may well fill its disks with debugging output.
The lines of the debugging output that are most useful for debugging load problems are:
SAFS_FetchStatus, Fid = 2003828163.77154.82248, Host 18.104.22.168 SRXAFS_FetchData, Fid = 2003828163.77154.82248
(The example above is partly truncated to highlight the interesting information). The Fid identifies the volume and inode within the volume; the volume is the first long number. So, for example, this was:
% vos examine 2003828163 pubsw.matlab61 2003828163 RW 1040060 K On-line afssvr5.Stanford.EDU /vicepa RWrite 2003828163 ROnly 2003828164 Backup 2003828165 MaxQuota 3000000 K Creation Mon Aug 6 16:40:55 2001 Last Update Tue Jul 30 19:00:25 2002 86181 accesses in the past day (i.e., vnode references) RWrite: 2003828163 ROnly: 2003828164 Backup: 2003828165 number of sites -> 3 server afssvr5.Stanford.EDU partition /vicepa RW Site server afssvr11.Stanford.EDU partition /vicepd RO Site server afssvr5.Stanford.EDU partition /vicepa RO Site
and from the Host information one can tell what system is accessing that volume.
Note that the output of vos_examine(1) also includes the access count, so once the problem has been identified, vos examine can be used to see if the access count is still increasing. Also remember that you can run vos examine on the read-only replica (e.g., pubsw.matlab61.readonly) to see the access counts on the read-only replica on all of the servers that it's located on.
This documentation is covered by the IBM Public License Version 1.0. It was converted from HTML to POD by software written by Chas Williams and Russ Allbery, based on work by Alf Wachsmann and Elizabeth Cassell.